当前位置： 首页 > wzjs >正文

如何开发网站建设业务邯郸旅游景点

wzjs 2025/9/19 5:34:25
如何开发网站建设业务,邯郸旅游景点,手机网站建设讯息,微信商户平台官网NLP 清晰版本查看 Sentence segmentation (split)Tokenisation (split)Named entity recognition (combine) 概念主要內容典型方法Distributional Semantics（分佈式語義）（分銷語義（分佈式語義）單詞的語義來自於它的…

NLP

清晰版本查看

  • Sentence segmentation (split)
  • Tokenisation (split)
  • Named entity recognition (combine)
概念主要內容典型方法
Distributional Semantics（分佈式語義）（分銷語義（分佈式語義）單詞的語義來自於它的上下文共現矩陣（Co-occurrence Matrix），PMI共現矩陣（共發生矩陣Chesi
Word Embeddings（詞嵌入）變成嵌入（詞嵌入）將單詞映射到低維向量空間，以捕捉語義關係Word2Vec、GloVe、FastText、BERTword2vec，手套，fasttx，bere
Word Sense Disambiguation（詞義消歧）單詞感官歧義（詞義消歧）根據語境確定詞的正確含義WordNet、機器學習、BERT

introduction to NLP

A research field focussed on creating software systems with knowledge about natural (human) language 研究重點是關於自認語言的知識

Interdisciplinary: makes use of theories from Linguistics 語言學理論, adopts an Engineering approach

Aimed at human-like understanding of language (but not yet there)

[!tip] Contributing disciplines
Linguistics: formal models of language, linguistic knowledge
Computer Science: representations, efficient processing, state machines, parsing
algorithms, probabilistic models, dynamic programming, machine learning
Mathematics: formal automata theory, computational modelling
Psychology: psychologically plausible modelling of language use

Many types of ambiguity歧義:

  • Phonological 語音學
    • multiple interpretations due to how it sounds 有些音聽起來一樣 那麼在識別的時候會存在很多種解釋
  • Lexical 詞匯
    • multiple interpretations due to a word having multiple senses 詞語的歧義，由於有的單詞本身帶有的意思多重導致的
  • Syntactic 句法
    • due to a word having more than one possible part of speech 一個單詞有多個演講部分
    • due to prepositional phrase attachment 介詞的附件
  • Semantic 語義
    • multiple possible interpretations解釋 unless knowledge of the world is available

[!danger] Two major approaches to NLP

  • Symbolic 象征
    • Rule- and dictionary-based systems 基於規則和字典系統
    • Captures linguistic knowledge in rules written by experts 補貨專家撰寫的規則中的語言知識
  • Statistical/Machine learning-based
    • Data-driven 數據驅動
    • Use of large amounts of (labelled) textual data (文本數據) to train systems, discover patterns

Comparison

SymbolicStatistical/Machine learning
✅  Expert knowledge yields highly precise results 專業知識會產生高度精確的結果✅Can generalise well on unseen examples可以很好的概括在看不見的例子上
❎Shortage of experts❎Need people for labelling
❎Laborious rule writing, dictionary preparation❎Time consuming and laborious labelling 耗時且費力
❎ Domain adaptation problematic域適應性問題❎Must retrain for new domain必須重新訓練
✅Results can be interpreted結果可以解釋❎ Often cannot inspect/change models通常無法檢查/更改模型
✅  Good when labelled data is hard to obtain當很難獲得標記的數據時✅ Good where dictionaries are unavailable

NLP Pipelines

A ‘complete’ NLP system is usually a pipeline of components

在这里插入图片描述

  • Sentence Segmentation 句子細分
  • Tokenisation 象征化
  • Parsing
  • Information

在这里插入图片描述

  • Sentence segmentation (split)
  • Tokenisation (split)
  • Named entity recognition (combine)

Why is NLP challenging:

  • Natural Language evolve:
    • new words appear constantly
    • Syntactic rules are flexible 句法規則靈活
    • ambiguity (模糊性) is inherent 固有的

Why Machine Learning for NLP

  • Traditional rule-based artificial intelligence (symbolic AI):
    • requires expert knowledge to engineer the rules 需要專家知識來設計
    • not flexible to adapt in multiple languages, domains, applications 不能靈活低使用與多種語言
  • Learning from data (machine learning) adapts:
    • to evolution: just learn from new data 從新數據中學習
    • to different applications: just learn with the appropriate target representation

Sentence Segmentation 句子細分

Sentence Detection is done before the text is tokenized 句子檢測在文本標記化之前進行

Task 順序：

  • Determination of boundaries between sentences確定句子之間的界限
  • Sentences used in subsequent NLP tasks 隨後NLP中使用的句子
  • Is it enough to detect the full stop
    • Could be an end-of-sentence (EOS) marker 可能是句子結束 (EOS) 標記
    • Or an end of abbreviation marker 縮寫標記的結尾
    • Or both

通常是Text Mining的第一步，因為它將非結構化文本拆分成基本處理單位

Variation in delimiters

  • Typical：“.”,“!”,“?”

[!todo] # Approaches

  • Regular expressions (Patterns) 正則表達式
  • Dictionaries (e.g., abbreviation lists) 字典
  • Hand-crafted rules 手工製作的規則(e.g., to check whether the word following an EOS delimiter starts with an uppercase character 檢查EOS界定符之後的單詞是否從大寫字符開始)
  • Statistical and ML approaches
  • Hybrid approaches 混合方法

Example of useful rules or Features

First character after potential EOS char 潛在EOS Char之後的第一個字符
● Should be uppercase? Problematic for some languages, e.g. German
● Permissible chars after potential EOS, e.g. lowercase characters?
Abbreviations 縮寫
● titles not likely to occur at EOS (e.g., Dr. Jones)
● company indicators could occur at EOS (e.g., MySocialMedia Inc.)

OpenNLP

The OpenNLP Sentence Detector cannot identify sentence boundaries based on the contents of the sentence 無法根據內容識別句子邊界

OpenNLP 開發文檔

輸出的是一個個string 每個string都是一個分解出來的句子

spaCy

for pdf and word docs

spaCy 開發文檔

Tokenisation 象征化

Why need Tokenisation

  • 機器學習模型通常使用「單詞」作為特徵，例如詞頻統計、情感分析等。
  • 英文單詞之間有空格，中文沒有，需要不同的處理方法。

Break sentence into tokens

3 main classes of tokens often considered

  • Morphosyntactic word
  • Punctuation mark or special symbol
  • A number
  • Endings of contractions, e.g., “'re” in “we’re”
  • Compounds and multi-words (e.g., daughter-in-law)

Challenges:

  • Character encoding
    • ASCII only?
    • Unicode (UTF-8)
  • ASCII-fication or romanisation of texts
    • Transliterations
  • Results from OCR may be poor
    • Pre-processing to detect/correct errors 預處理
    • OCR errors may appear as correct text (but not intended text)

[!attention] Challenging

  • Hyphenation
    • Manchester-based
    • Sister-in-law
  • Telephone numbers: many different Formats
    • with whitespace, dots
    • slashes, hyphens, parentheses, plus signs

[!example]+

  • Dates: 04 January 2018; 04-01-2018, Jan 4, 2018
  • Decimals: 0.05; 3.4; .6
  • Monetary values: a £5-a-dish dinner

Split or not to split

  • Sentence segmentation (split)
  • Tokenisation (split)
  • Named entity recognition (combine)
    In other words: tokenisation is knowing when to split (not when to combine)

Annotation Formats

Annotation（標註） 是在文本中為每個 token 添加額外的信息，例如詞性標註（Part-of-Speech Tagging, POS Tagging）、命名實體識別（Named Entity Recognition, NER）等。

Understanding documents

Documents rarely have a simple structure 很少有簡單結構
Documents are meant to be human-readable

[!example]

  • news article
  • research article

Annotations:

Enabling machine-readablility

在这里插入图片描述

在这里插入图片描述
如圖展示的 他從單純的文字變成機器可讀的樣式

Types of annotation Formats

Boundary notation 邊界符號
Inline markup language elements 內聯標記語言元素
Stand-off

  • delimiter-separated values (DSV)
  • JSON

Part-od-Speech Tagging (POS Tagging) 詞性標註

給每個單詞分配詞性標籤

[!example]

import nltk 
nltk.download('averaged_perceptron_tagger') 
sentence = "She runs fast." 
tokens = word_tokenize(sentence) 
pos_tags = nltk.pos_tag(tokens) 
print(pos_tags) 
# [('She', 'PRP'), ('runs', 'VBZ'), ('fast', 'RB')]

Named Entity Recpgnition (NER) 命名實體識別

NER 用來識別文本中的人名、地名、組織名等專有名詞

句子：
“Apple is looking at buying U.K. startup for $1 billion.”

NER 標註結果：

  1. “Apple” → ORG（組織）
  2. “U.K.” → GPE（地名）
  3. “$1 billion” → MONEY（貨幣）
import spacy 
nlp = spacy.load("en_core_web_sm") 
sentence = "Apple is looking at buying U.K. startup for $1 billion." 
doc = nlp(sentence) for ent in doc.ents: print(ent.text, ent.label_) 
# Apple ORG 
# U.K. GPE# $1 billion MONEY

Boundary Notation

Done at the level of individual tokens 在單個token完成
How do we encode units of interest spanning several tokens
BIO: B=Begin I=Inside O=Outside

[!example]
在这里插入图片描述

  • Strengths
    • simple
  • limitations
    • cannot handle hierarchical or structured annotations e.g., nested entities(NES) 嵌套實體, relations events

[!example] Nested entities

在这里插入图片描述

展示了Named Entity Recognition (NER) 命名實體識別的結果

  • 是一種NLP技術，用來識別文本中的關鍵實體，比如地名 (GPE)，組織 (Org), 人物 (Person) 和 事件 (Conflict_Attack)
  • GPE：
    • 國家或地區名稱
  • Org：
    • 機構或組織的名稱
  • Person：
    • 具體人名
  • Contact_Meet：
    • 這類標籤標識涉及會議或高級別會議
  • Conflict_Attack（衝突/攻擊, Conflict_Attack
    • 標識與戰爭或攻擊有關的事件
  • 關係標註（箭頭）
    • 表示某人參與了一場活動
    • Target (目標)
    • 標識某個實體是某個行動的目標

Inline markup language elements

By addition of markup tags within text

  • HTML
  • XML

[!example]
在这里插入图片描述

Strengths:

  • can handle annotations which are hierarchical (e.g., nested NEs, trees) and structured (e.g., events) 可以處理分層的注釋
    Limitations:
  • requires substantial processing with standard XML parsers 對標準XML處理器進行大量處理
  • impossible to encode overlapping/intersecting annotations, e.g., second Iraqi city of Basra 無法編碼重疊/相交注釋

Stand-off Annotations (JSON格式)

將標註信息與原始文本分離的標註方式，而不是將標註直接嵌入到文本內部。

為什麼使用 Stand-off Annotation？

  1. 避免污染原始數據：原始文本保持不變，標註信息存儲在外部文件或數據結構中。
  2. 允許多層次標註：可以為相同文本提供多種標註（如詞性、語法結構、命名實體等），並獨立管理它們。
  3. 便於版本控制：標註數據和原始文本分開存儲，有助於管理不同版本的標註信息。
  4. 支持長文本處理：對於超大文本，標註信息存儲在索引數據結構中，提高效率。

[!example]+

UK and US discuss the role of UN.

stand-off 標註文件

{"text": "UK and US discuss the role of UN.","annotations": [{"id": "T1","type": "GPE","start": 0,"end": 2,"text": "UK"},{"id": "T2","type": "GPE","start": 7,"end": 9,"text": "US"},{"id": "T3","type": "ORG","start": 27,"end": 29,"text": "UN"}]
}

annotations are stored separately
requires a way to link between annotations and text
links annotations to text using indexing based on character offsets (computed over raw
text)

  • strength:
    • original raw text is left untouched 原始文本未觸及
    • can handle structured and overlapping annotations 可以處理結構化和重疊的注釋
  • limitations:
    • not readily human-readable
http://www.dtcms.com/wzjs/788245.html

相关文章：

  • 谷歌网站推广排名工具网站建设要注意那些问题
  • jz做网站做个商城网站怎么做便宜吗
  • 网站架构模板wordpress 全局音乐
  • 上海做网站站优云一一十七北京服装设计公司前十名
  • 安徽seo网站网站内网页标题对百度排名
  • 网站忘记密码功能成都网站建设桔子科技
  • 自建网站平台有哪些wordpress美化登录
  • 郑州龙华小学网站建设WordPress博客Modown模板
  • 网站建设合同书wordpress国内支付
  • 网站建设费用贵不贵做企业网站有效果吗
  • ECMS做的网站做死活题网站
  • 杭州网站程序开发公司网页设计实训结论
  • 网站单页面怎么做的汝阳网站建设
  • 做ppt常用的网站有哪些东莞网站设计建设公司
  • 禹州市门户网站建设华为最新消息
  • 网站空间免泰安市人才交流服务中心
  • 盐城公司网站建设做网站最基础需要什么条件
  • 贵阳网站推广￥做下拉去118cr五大建设内容
  • 国外网站开发技术福州网站建设网络公司
  • 网站制作公司兴田德润i在哪里公司网站建设要求书
  • 杭州自助建站河北建设厅网站开通账号
  • 网页设计与网站建设西方设计网站
  • 网站建设工程师职责在线一键生成网页
  • 在门户网站做产品seo广州电商小程序开发
  • 域名做网站北海建设网站
  • 万和城官方网站北京流感最新消息
  • 延吉市建设厅网站卖狗做网站什么关键词最好
  • 跨境电商网站开发公司现在广告行业好做吗
  • 莒县做网站的电话flash网站的优点和缺点
  • 网站建设体会doc贵阳官网建设价格