Appreciate life and record it

Dragonfly

 1769  4    
Dragonfly

Dragonfly

介绍

用于类似于cdn分发的一个阿里云开源组件，可以分发镜像或者较大文件，利用 内网带宽。官网https://d7y.io/zh/。 感兴趣可以看看这个视频

install 

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

root@master0:~# git clone https://github.com/dragonflyoss/Dragonfly2.git
Cloning into 'Dragonfly2'...
remote: Enumerating objects: 22270, done.
remote: Counting objects: 100% (70/70), done.
remote: Compressing objects: 100% (64/64), done.
remote: Total 22270 (delta 17), reused 22 (delta 4), pack-reused 22200
Receiving objects: 100% (22270/22270), 10.85 MiB | 192.00 KiB/s, done.
Resolving deltas: 100% (14254/14254), done.
root@master0:~/Dragonfly2/deploy/docker-compose# export IP="192.168.122.82"
root@master0:~/Dragonfly2/deploy/docker-compose# ./run.sh
docker: 'compose' is not a docker command.
See 'docker --help'
redis is up-to-date
mysql is up-to-date
Recreating manager ... done
Creating scheduler ... done
Creating seed-peer ... done
Creating peer      ... done
wait for all service ready: 3/6,0 times check
wait for all service ready: 5/6,1 times check
  Name                 Command                  State                               Ports                         
------------------------------------------------------------------------------------------------------------------
manager     /opt/dragonfly/bin/server        Up (healthy)   0.0.0.0:65003->65003/tcp, 0.0.0.0:8090->8080/tcp      
mysql       docker-entrypoint.sh mariadbd    Up (healthy)   0.0.0.0:3306->3306/tcp                                
peer        /opt/dragonfly/bin/dfget d ...   Up (healthy)   0.0.0.0:65000->65000/tcp, 0.0.0.0:65001->65001/tcp,   
                                                            0.0.0.0:65002->65002/tcp                              
redis       docker-entrypoint.sh --req ...   Up (healthy)   0.0.0.0:6379->6379/tcp                                
scheduler   /opt/dragonfly/bin/scheduler     Up (healthy)   0.0.0.0:8002->8002/tcp                                
seed-peer   /opt/dragonfly/bin/dfget d ...   Up (healthy)   65001/tcp, 0.0.0.0:65006->65006/tcp,                  
                                                            0.0.0.0:65007->65007/tcp, 0.0.0.0:65008->65008/tcp

访问IP:8090即可，默认是8080 ../images/dragonfly/welcome.png◎ ../images/dragonfly/welcome.png

username is root and password is dragonfly.

点击next， create seed ../images/dragonfly/create-seed.png◎ ../images/dragonfly/create-seed.png

The created seed peer cluster id is 1. When deploying a seed peer instance, you need to report the seed peer cluster id associated with the seed peer instance, and the manager service address.

For details, refer to the two fields manager.addr and manager.seedPeerClusterID in the seed peer configuration.

这里提示是seed peer（用于各个客户端初始下载的）需要在绑定的时候需要设 置seed cluster集群id为1 主要改manager.addr和manager.seedPeerClusterID

继续点击next，开始创建scheduler ../images/dragonfly/create-scheduler.png◎ ../images/dragonfly/create-scheduler.png

Create Scheduler Cluster

A scheduler cluster needs to be created in a P2P network.

Seed peer instances in the seed peer cluster associated with the scheduler cluster will become the download root node in the P2P network. The seed peer cluster field in the scheduler cluster form is the seed peer cluster associated with the scheduler clsuter in the P2P network.

The created scheduler cluster id is 1. When deploying a scheduler instance, you need to report the scheduler cluster id associated with the scheduler instance, and the manager service address. For details, refer to the two fields manager.addr andmanager.schedulerClusterID in thescheduler configuration.

也就相当于cluster需要一个scheduler，这个负责告诉各个客户端从哪下载。设 置字段manager.addr和manager.schedulerClusterID。

继续点击下一步发现提示创建好了，如图 ../images/dragonfly/create-cluster.png◎ ../images/dragonfly/create-cluster.png

可以发现daemon相当于客户端安装的，从对应的seed peer去获取资源，seed peer如果没有会去远程拉，比如一个镜像或者一个包或者共享存储等等。

用户可以调用daemon dfget去下载，下载可能是其他peer或者seed peer或者代理 下载或者调用三方sdk，最终注册上去

下载文件

https://github.com/dragonflyoss/Dragonfly/blob/master/docs/user_guide/download_files.md

https://github.com/dragonflyoss/Dragonfly/blob/master/docs/design/design.md

去docker的peer节点执行dfget -u=xxx

dfget is the client of dragonfly which takes a role of peer in a P2P network. When user triggers a file downloading task, dfget will download the pieces of file from other peers. Meanwhile, it will act as an uploader to support other peers to download pieces from it if it owns them. In addition, dfget has the abilities to provide more advanced functionality, such as network bandwidth limit, transmission encryption and so on.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

root@master0:~# docker ps
CONTAINER ID   IMAGE                           COMMAND                  CREATED        STATUS                                     PORTS                                             NAMES
e1514f17c574   dragonflyoss/dfdaemon:latest    "/opt/dragonfly/bin/…"   18 hours ago   Up 6 seconds (healthy)                     0.0.0.0:65000-65002->65000-65002/tcp              peer
c5c9b771eea3   dragonflyoss/dfdaemon:latest    "/opt/dragonfly/bin/…"   18 hours ago   Up Less than a second (health: starting)   65001/tcp, 0.0.0.0:65006-65008->65006-65008/tcp   seed-peer
080589c9077f   dragonflyoss/scheduler:latest   "/opt/dragonfly/bin/…"   18 hours ago   Up Less than a second (health: starting)   0.0.0.0:8002->8002/tcp                            scheduler
f480d8315c2f   dragonflyoss/manager:latest     "/opt/dragonfly/bin/…"   18 hours ago   Restarting (1) 1 second ago                                                                  manager

root@master0:~# docker exec -it e1514f17c574 /bin/sh

/ # dfget -u=https://blog.liuliancao.com
--2023-01-17 02:06:55--  https://blog.liuliancao.com
dfget version: v2.0.8
current user: root, default peer ip: 172.21.0.3
output path: /blog.liuliancao.com
init success and start to download
finish total length 43878 bytes
download success: true cost: 30098 ms
/ # dfget -u=https://blog.liuliancao.com
--2023-01-17 02:39:01--  https://blog.liuliancao.com
dfget version: v2.0.8
current user: root, default peer ip: 172.21.0.3
output path: /blog.liuliancao.com
init success and start to download
finish total length 43878 bytes
download success: true cost: 4 ms
/ # dfget -u=https://blog.liuliancao.com
--2023-01-17 02:39:03--  https://blog.liuliancao.com
dfget version: v2.0.8
current user: root, default peer ip: 172.21.0.3
output path: /blog.liuliancao.com
init success and start to download
finish total length 43878 bytes
download success: true cost: 2 ms

关于dfget更多请参考dfget git

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144

/opt/dragonfly # cat /etc/dragonfly/dfget.yaml |grep -v '^\s*#'|grep -v '^$'
aliveTime: 0s
gcInterval: 1m0s
workHome: ''
logDir: ''
cacheDir: ''
pluginDir: ''
dataDir: ''
keepStorage: true
console: false
verbose: true
pprof-port: -1
jaeger: ""
scheduler:
  manager:
    enable: false
  scheduleTimeout: 30s
  disableAutoBackSource: false
  netAddrs:
    - type: tcp
      addr: 192.168.122.82:8002
host:
  advertiseIP: 192.168.122.82
  location: ""
  idc: ""
  securityDomain: ""
  netTopology: ""
download:
  concurrent:
    thresholdSize: 10M
    thresholdSpeed: 2M
    goroutineCount: 4
    initBackoff: 0.5
    maxBackoff: 3
    maxAttempts: 3
  calculateDigest: true
  totalRateLimit: 1024Mi
  perPeerRateLimit: 512Mi
  trafficShaperType: sampling
  pieceDownloadTimeout: 30s
  prefetch: false
  transportOption:
    dialTimeout: 2s
    keepAlive: 30s
    maxIdleConns: 100
    idleConnTimeout: 90s
    responseHeaderTimeout: 2s
    tlsHandshakeTimeout: 1s
    expectContinueTimeout: 2s
  downloadGRPC:
    security:
      insecure: true
      cacert: ''
      cert: ''
      key: ''
      tlsVerify: true
      tlsConfig: null
    unixListen:
      socket: ''
  peerGRPC:
    security:
      insecure: true
      cacert: ''
      cert: ''
      key: ''
      tlsVerify: true
    tcpListen:
      port: 65000
upload:
  rateLimit: 1024Mi
  security:
    insecure: true
    cacert: ''
    cert: ''
    key: ''
    tlsVerify: false
  tcpListen:
    port: 65002
objectStorage:
  enable: false
  filter: 'Expires&Signature&ns'
  maxReplicas: 3
  security:
    insecure: true
    tlsVerify: true
  tcpListen:
    port: 65004
storage:
  taskExpireTime: 6h
  strategy: io.d7y.storage.v2.simple
  diskGCThreshold: 50Gi
  diskGCThresholdPercent: 80
  multiplex: true
health:
  security:
    insecure: true
    cacert: ''
    cert: ''
    key: ''
    tlsVerify: false
  tcpListen:
    port: 40901
proxy:
  defaultFilter: "Expires&Signature&ns"
  defaultTag: ''
  security:
    insecure: true
    cacert: ''
    cert: ''
    key: ''
    tlsVerify: false
  tcpListen:
    namespace: ''
    port: 65001
  registryMirror:
    dynamic: true
    url: https://index.docker.io
    insecure: true
    certs: []
    direct: false
    useProxies: false
  proxies:
    - regx: blobs/sha256.*
    - regx: file-server.*
    - regx: some-registry/
      useHTTPS: true
    - regx: no-proxy-reg
      direct: true
    - regx: some-registry
      redirect: another-registry
    - regx: ^http://some-registry/(.*)
      redirect: http://another-registry/$1
  hijackHTTPS:
    cert: ""
    key: ""
    hosts:
      - regx: mirror.aliyuncs.com:443 # regexp to match request hosts
        insecure: true
        certs: []
  maxConcurrency: 0
  whiteList:
    - host: ""
      regx: ".*"
      ports:

主要指定了scheduler地址

可以发现服务器上使用dfget会触发下载任务，任务触发以后再次下载就是从内 网了，这个办法理论上可以应用于包集群或者linux服务器。

参考文档