Skip to content
.NET 开发者指北.NET 开发者指北
CMS
.NET指北
FreeKit
Docker
关于
博客
github icon
    • 开发起步
      • Newtonsoft.Json基础问题
        • 依赖注入scrutor
          • 认证与授权
            • 获取控制器及方法特性标签
              • 认证鉴权状态
                • 全局敏感词处理
                  • 1.ToolGood.Words
                    • 2. 循环使用Replace
                      • FreeSql全局处理敏感词
                      • GitHub第三方授权登录
                        • QQ第三方授权登录
                          • CAP实现EventBus
                            • Scriban 模板语言

                            全局敏感词处理

                            calendar icon2020年2月28日timer icon大约 2 分钟word icon约 735 字

                            此页内容
                            • 1.ToolGood.Words
                            • 2. 循环使用Replace
                            • FreeSql全局处理敏感词

                            # 全局敏感词处理

                            基于ToolGood.Words类库,配合敏感字的文本文件,写的API接口。

                            • https://github.com/luoyunchong/dotnetcore-examples/tree/master/aspnetcore-stopwordsopen in new window

                            一共二种方式

                            # 1.ToolGood.Words

                            类库配合敏感库

                            • ToolGood.Wordsopen in new window

                            简单用法

                            [Fact]
                            public void IssuesTest_17()
                            {
                                var illegalWordsSearch = new IllegalWordsSearch();
                                string s = "中国|zg人|abc";
                                illegalWordsSearch.SetKeywords(s.Split('|'));
                                var str = illegalWordsSearch.Replace("我是中美国人厉害中国完美abcddb好的", '*');
                            
                                Assert.Equal("我是中美国人厉害**完美***ddb好的", str);
                            }
                            
                            1
                            2
                            3
                            4
                            5
                            6
                            7
                            8
                            9
                            10

                            配合敏感库文本文件,写的工具类

                            二个文件放到wwwroot/_Illegal目录下,通过ReplaceStopWords方法调用即可对

                            • https://github.com/toolgood/ToolGood.Words/blob/master/csharp/ToolGood.Words.Test/_Illegal/IllegalKeywords.txtopen in new window
                            • https://github.com/toolgood/ToolGood.Words/blob/master/csharp/ToolGood.Words.Test/_Illegal/IllegalUrls.txtopen in new window
                            public class ToolGoodUtils
                            {
                                //敏感库只要这二个文件存在即可
                                //本地敏感库缓存-https://github.com/toolgood/ToolGood.Words/tree/master/csharp/ToolGood.Words.Test/_Illegal
                                //因为需要上传至github并同步gitee,安全起见,所以未上传至git,需要自行下载并复制
                                private const string KeywordsPath = "wwwroot/_Illegal/IllegalKeywords.txt";
                                private const string UrlsPath = "wwwroot/_Illegal/IllegalUrls.txt";
                            
                                private const string InfoPath = "wwwroot/_Illegal/IllegalInfo.txt";
                                private const string BitPath = "wwwroot/_Illegal/IllegalBit.iws";
                            
                                private static IllegalWordsSearch _search;
                                /// <summary>
                                /// 本地敏感库,文件修改后,重新创建缓存Bit
                                /// </summary>
                                /// <returns></returns>
                                public static IllegalWordsSearch GetIllegalWordsSearch()
                                {
                                    if (_search == null)
                                    {
                                        string ipath = Path.GetFullPath(InfoPath);
                                        if (File.Exists(ipath) == false)
                                        {
                                            _search = CreateIllegalWordsSearch();
                                        }
                                        else
                                        {
                                            var texts = File.ReadAllText(ipath).Split('|');
                                            if (new FileInfo(Path.GetFullPath(KeywordsPath)).LastWriteTime.ToString("yyyy-MM-dd HH:mm:ss") !=
                                                texts[0] ||
                                                new FileInfo(Path.GetFullPath(UrlsPath)).LastWriteTime.ToString("yyyy-MM-dd HH:mm:ss") !=
                                                texts[1]
                                            )
                                            {
                                                _search = CreateIllegalWordsSearch();
                                            }
                                            else
                                            {
                                                var s = new IllegalWordsSearch();
                                                s.Load(Path.GetFullPath(BitPath));
                                                _search = s;
                                            }
                                        }
                                    }
                                    return _search;
                                }
                            
                                private static IllegalWordsSearch CreateIllegalWordsSearch()
                                {
                                    string[] words1 = File.ReadAllLines(Path.GetFullPath(KeywordsPath), Encoding.UTF8);
                                    string[] words2 = File.ReadAllLines(Path.GetFullPath(UrlsPath), Encoding.UTF8);
                                    var words = new List<string>();
                                    foreach (var item in words1)
                                    {
                                        words.Add(item.Trim());
                                    }
                                    foreach (var item in words2)
                                    {
                                        words.Add(item.Trim());
                                    }
                            
                                    var search = new IllegalWordsSearch();
                                    search.SetKeywords(words);
                            
                                    search.Save(Path.GetFullPath(BitPath));
                            
                                    var text = new FileInfo(Path.GetFullPath(KeywordsPath)).LastWriteTime.ToString("yyyy-MM-dd HH:mm:ss") + "|"+ new FileInfo(Path.GetFullPath(UrlsPath)).LastWriteTime.ToString("yyyy-MM-dd HH:mm:ss");
                                    File.WriteAllText(Path.GetFullPath(InfoPath), text);
                            
                                    return search;
                                }
                            }
                            
                            1
                            2
                            3
                            4
                            5
                            6
                            7
                            8
                            9
                            10
                            11
                            12
                            13
                            14
                            15
                            16
                            17
                            18
                            19
                            20
                            21
                            22
                            23
                            24
                            25
                            26
                            27
                            28
                            29
                            30
                            31
                            32
                            33
                            34
                            35
                            36
                            37
                            38
                            39
                            40
                            41
                            42
                            43
                            44
                            45
                            46
                            47
                            48
                            49
                            50
                            51
                            52
                            53
                            54
                            55
                            56
                            57
                            58
                            59
                            60
                            61
                            62
                            63
                            64
                            65
                            66
                            67
                            68
                            69
                            70
                            71
                            72

                            # 2. 循环使用Replace

                            方案:通过维护敏感库,循环replace 大佬分享给我的,稍微改成了从文件中获取敏感字。

                            public static class StopWords
                            {
                            
                                static readonly ConcurrentDictionary<string, bool> FunNlpDataSensitive = new ConcurrentDictionary<string, bool>();
                                static readonly ConcurrentDictionary<int, string> ReplaceNewValue = new ConcurrentDictionary<int, string>();
                            
                                private const string KeywordsPath = "wwwroot/_Illegal/IllegalKeywords.txt";
                                private const string UrlsPath = "wwwroot/_Illegal/IllegalUrls.txt";
                            
                            
                                static StopWords()
                                {
                                    LoadDataFromFile();
                                }
                            
                                public static void LoadDataFromFile()
                                {
                                    string words1 = File.ReadAllText(Path.GetFullPath(KeywordsPath), Encoding.UTF8);
                                    string words2 = File.ReadAllText(Path.GetFullPath(UrlsPath), Encoding.UTF8);
                                    LoadDataFromText(words1);
                                    LoadDataFromText(words2);
                                }
                            
                            
                                public static void LoadDataFromText(string text)
                                {
                                    int oldcount = FunNlpDataSensitive.Count;
                                    foreach (string wd in text.Split('\n'))
                                    {
                                        string keykey = wd.Trim().Trim('\r', '\n').Trim();
                                        if (string.IsNullOrEmpty(keykey)) continue;
                                        FunNlpDataSensitive.TryAdd(keykey, true);
                                        if (ReplaceNewValue.ContainsKey(keykey.Length) == false)
                                            ReplaceNewValue.TryAdd(keykey.Length, "".PadRight(keykey.Length, '*'));
                                    }
                                    Console.WriteLine($"敏感词加载完毕,增加数量:{FunNlpDataSensitive.Count - oldcount}");
                                }
                            
                            
                                /// <summary>
                                /// 替换所有敏感词为 *
                                /// </summary>
                                /// <param name="that"></param>
                                /// <returns></returns>
                                public static string ReplaceStopWords(this string that)
                                {
                                    foreach (var wd in FunNlpDataSensitive.Keys)
                                        that = that.Replace(wd, ReplaceNewValue.TryGetValue(wd.Length, out var tryval) ? tryval : "".PadRight(wd.Length, '*'));
                                    return that;
                                }
                            }
                            
                            1
                            2
                            3
                            4
                            5
                            6
                            7
                            8
                            9
                            10
                            11
                            12
                            13
                            14
                            15
                            16
                            17
                            18
                            19
                            20
                            21
                            22
                            23
                            24
                            25
                            26
                            27
                            28
                            29
                            30
                            31
                            32
                            33
                            34
                            35
                            36
                            37
                            38
                            39
                            40
                            41
                            42
                            43
                            44
                            45
                            46
                            47
                            48
                            49
                            50
                            51

                            # FreeSql全局处理敏感词

                            使用FreeSql这个ORM时,全局处理string类型的值,进行敏感词处理。代码在StartUp.cs的构造函数中。

                            //敏感词处理
                            IllegalWordsSearch illegalWords = ToolGoodUtils.GetIllegalWordsSearch();
                            
                            Fsql.Aop.AuditValue += (s, e) =>
                            {
                                if (e.Column.CsType == typeof(string) && e.Value != null)
                                {
                                    string oldVal = (string)e.Value;
                                    string newVal = illegalWords.Replace(oldVal);
                                    //第二种处理敏感词的方式
                                    //string newVal = oldVal.ReplaceStopWords();
                                    if (newVal != oldVal)
                                    {
                                        e.Value = newVal;
                                    }
                                }
                            };
                            
                            1
                            2
                            3
                            4
                            5
                            6
                            7
                            8
                            9
                            10
                            11
                            12
                            13
                            14
                            15
                            16
                            17
                            edit icon在 GitHub 上编辑此页open in new window
                            上次编辑于: 2020/7/24 17:10:08
                            贡献者: luoyunchong
                            上一页
                            认证鉴权状态
                            下一页
                            GitHub第三方授权登录
                            MIT Licensed | Copyright © 2021-present luoyunchong
                            苏ICP备16046457号-1

                            该应用可以安装在你的 PC 或移动设备上。这将使该 Web 应用程序外观和行为与其他应用程序相同。它将在出现在应用程序列表中,并可以固定到主屏幕,开始菜单或任务栏。此 Web 应用程序还将能够与其他应用程序和你的操作系统安全地进行交互。

                            详情