Python自动化运维

自动化运维平台CMDB介绍、需求分析及核心代码的实现。

CMDB介绍

谈到自动化运维,不得不提CMDB。

CMDB – Configuration Management Database,即配置管理数据库, CMDB存储与管理企业IT架构中设备的各种配置信息,它与所有服务支持和服务交付流程都紧密相联,支持这些流程的运转、发挥配置信息的价值,同时依赖于相关流程保证数据的准确性。

实际项目中,70%~80%的IT相关问题与环境的变更有着直接的关系。实施变更管理的难点和重点并不是工具,而是流程。即通过一个自动化的、可重复的流程管理变更,使得当变更发生的时候,有一个标准化的流程去执行,能够预测到这个变更对整个系统管理产生的影响,并对这些影响进行评估和控制。而变更管理流程自动化的实现关键就是CMDB。

CMDB工具中至少包含这几种关键的功能:整合、调和、同步、映射和可视化。

  • 整合是指能够充分利用来自其他数据源的信息,对CMDB中包含的记录源属性进行存取,将多个数据源合并至一个视图中,生成连同来自CMDB和其他数据源信息在内的报告;
  • 调和能力是指通过对来自每个数据源的匹配字段进行对比,保证CMDB中的记录在多个数据源中没有重复现象,维持CMDB中每个配置项目数据源的完整性;自动调整流程使得初始实施、数据库管理员的手动运作和现场维护支持工作降至最低;
  • 同步指确保CMDB中的信息能够反映联合数据源的更新情况,在联合数据源更新频率的基础上确定CMDB更新日程,按照经过批准的变更来更新 CMDB,找出未被批准的变更;
  • 应用映射与可视化,说明应用间的关系并反应应用和其他组件之间的依存关系,了解变更造成的影响并帮助诊断问题。

Devops工程入门引导

开发 + 运维 = DevOps,这是当前最火热的互联网IT理念。当然实际构建一个完整且功能完善的Devops系统远比想象中的复杂困难。

但是这一理念起码从侧面反映了要构建这一系统,必须要会一门开发语言,就当前编程语言的热度与快速上手,非python莫属。

自动化运维难点和痛点

开发人员:没有系统管理、网络管理等相关运维工作经验,项目设计往往是大打折扣。

运维人员:不具备开发能力,没有项目的开发经验或能力。

因此作为一个好的Devops,至少需要2年以上的运维+开发工作经验。

资产管理需求

  • 存储所有IT资产信息
  • 数据可手动添加
  • 硬件信息可自动收集
  • 硬件信息可自动变更

定义表结构

  • 各种硬件都能存
  • 资产变更有纪录
  • 资产ID永不变
  • 资产要有状态机

入门引导

后端部分实现如下:

  • 自动化资产扫描发现
  • ansible自动化任务

核心知识点如下:

整体工程设计:

  • 资产自动化扫描发现

    用python程序扫描发现企业内部的所有资产,当资产发生变动能自动及时发现并完成资产变更。

  • ansible自动化任务执行

    使用ansible的ad-hoc和playbook实现批量主机的自动化任务。

    (注:S - Server,C - Client)

Python自动化运维

model层设计

from django.db import models
 
# Create your models here.
from Wolf.models import UserProfile
 
class Asset(models.Model):
    asset_type_choices = (
        ('server', u'服务器'),
        ('switch', u'交换机'),
        ('router', u'路由器'),
        ('firewall', u'防火墙'),
        ('storage', u'存储设备'),
        ('NLB', u'NetScaler'),
        ('wireless', u'无线AP'),
        ('software', u'软件资产'),
        ('others', u'其它类'),
    )
    asset_type = models.CharField(choices=asset_type_choices,max_length=64, default='server')
    name = models.CharField(max_length=64,unique=True)
    sn = models.CharField(u'资产SN号',max_length=128, unique=True)
    manufactory = models.ForeignKey('Manufactory',verbose_name=u'制造商',null=True, blank=True)
    management_ip = models.GenericIPAddressField(u'管理IP',blank=True,null=True)
    contract = models.ForeignKey('Contract',verbose_name=u'合同',blank=True,null=True)
    trade_date = models.DateField(u'购买时间',null=True,blank=True)
    expire_date = models.DateField(u'过保修期',null=True,blank=True)
    price = models.FloatField(u'价格',null=True,blank=True)
    business_unit = models.ForeignKey('BusinessUnit',verbose_name=u'所属业务线',null=True,blank=True)
 
    tags = models.ManyToManyField('Tag',blank=True)
    admin = models.ForeignKey(UserProfile,verbose_name=u'资产管理员',null=True, blank=True)
    idc = models.ForeignKey('IDC',verbose_name=u'IDC机房',null=True, blank=True)
    memo = models.TextField(u'备注',null=True,blank=True)
    create_date = models.DateTimeField(blank=True,auto_now_add=True)
    update_date = models.DateTimeField(blank=True,auto_now=True)
    
    class Meta:
        verbose_name = '资产总表'
        verbose_name_plural = '资产总表'
        
    def __str__(self):
        return 'id:%s name:%s' %(self.id,self.name)
    
    
class Server(models.Model):
    asset = models.OneToOneField('Asset')
    created_by_choices = (
        ('auto','Auto'),
        ('manual','Manual'),
    )
    create_by = models.CharField(max_length=32,choices=created_by_choices,default='auto')
    hosted_on = models.ForeignKey('self',related_name='hosted_on_server',blank=True,null=True)
    model = models.CharField(u'型号',max_length=128,null=True,blank=True)
    raid_type = models.CharField(u'raid类型',max_length=512,blank=True,null=True)
    os_type = models.CharField(u'操作系统类型',max_length=64,blank=True,null=True)
    os_distribution = models.CharField(u'发行版本',max_length=64,blank=True,null=True)
    os_release = models.CharField(u'操作系统版本',max_length=64,blank=True,null=True)
 
    create_date = models.DateTimeField(blank=True,auto_now_add=True)
    update_date = models.DateTimeField(blank=True,auto_now=True)
    class Meta:
        verbose_name = '服务器'
        verbose_name_plural = '服务器'
 
    def __str__(self):
        return '%s sn:%s' %(self.asset,self.asset.sn)
    
    
class NetworkDevice(models.Model):
    asset = models.ForeignKey('Asset')
    vlan_ip = models.GenericIPAddressField(u'VlanIP',blank=True,null=True)
    intranet_ip = models.GenericIPAddressField(u'内网IP',blank=True,null=True)
    sn = models.CharField(u'SN号',max_length=128,unique=True)
    model = models.CharField(u'型号',max_length=128,null=True,blank=True)
    firmware = models.ForeignKey('Software',blank=True,null=True)
    port_num = models.SmallIntegerField(u'端口个数',null=True,blank=True)
    device_detail = models.TextField(u'设置详细配置',null=True,blank=True)
    create_date = models.DateTimeField(auto_now_add=True)
    update_date =models.DateTimeField(auto_now=True)
 
    class Meta:
        verbose_name = '网络设备'
        verbose_name_plural = '网络设备'
 

class Software(models.Model):
    os_type_choices = (
        ('linux','Linux'),
        ('windows','Windows'),
        ('network_firmware','Network Firmware'),
        ('software','Softwares')
    )
 
    os_distribution_choices = (
        ('windows','Windows'),
        ('centos','CentOS'),
        ('ubuntu','Ubuntu')
    )
    
    type = models.CharField(u'系统类型',choices=os_type_choices,max_length=64,help_text=u'eg. GNU/Linux',default=1)
    distribution = models.CharField(u'发行版本',choices=os_distribution_choices,max_length=32,default='windows')
    version = models.CharField(u'软件/系统版本',max_length=64,help_text=u'eg. CentOS release 6.5 (Final)', unique=True)
    language_choices = (
        ('cn',u'中文'),
        ('en',u'英文'),
    )
    language = models.CharField(u'系统语言',choices=language_choices,max_length=64,default='cn')
 
    def __str__(self):
 
        return self.version
 
    class Meta:
        verbose_name = '软件/系统'
        verbose_name_plural = '软件/系统'
        
 
class CPU(models.Model):
    asset = models.OneToOneField('Asset')
    cpu_model = models.CharField(u'CPU型号',max_length=128,blank=True)
    cpu_count = models.SmallIntegerField(u'物理cpu个数')
    cpu_core_count = models.SmallIntegerField(u'cpu核数')
    memo = models.TextField(u'备注',null=True,blank=True)
    create_date = models.DateTimeField(auto_now_add=True)
    update_date =models.DateTimeField(auto_now=True,blank=True,null=True)
 
    class Meta:
        verbose_name = 'CPU部件'
        verbose_name_plural = 'CPU部件'
    def __str__(self):
        return self.cpu_model
    
    
class RAM(models.Model):
    asset = models.ForeignKey('Asset')
    sn = models.CharField(u'SN号',max_length=128,blank=True,null=True)
    manufactory = models.CharField(u'制造商',max_length=64,blank=True,null=True)
    model = models.CharField(u'内存类型',max_length=128)
    slot = models.CharField(u'插槽',max_length=64)
    capacity = models.IntegerField(u'内存大小(MB)')
    memo = models.CharField(u'备注',max_length=128,blank=True,null=True)
    create_date = models.DateTimeField(blank=True,auto_now_add=True)
    update_date =models.DateTimeField(auto_now=True,blank=True,null=True)
 
    def __str__(self):
        return '%s:%s:%s' %(self.asset_id,self.slot,self.capacity)
    
    class Meta:
        verbose_name = 'RAM'
        verbose_name_plural = 'RAM'
        unique_together = ['asset','model',]
    auto_create_fields = ['sn','slot','model','capacity',]
 

class Disk(models.Model):
    asset = models.ForeignKey('Asset')
    sn = models.CharField(u'SN号',max_length=128,blank=True,null=True)
    slot = models.CharField(u'插槽位',max_length=64)
    manufactory = models.CharField(u'制造商',max_length=64,blank=True,null=True)
    model = models.CharField(u'磁盘型号',max_length=128,blank=True,null=True)
    capacity = models.FloatField(u'磁盘容量GB')
    disk_iface_choice = (
        ('SATA','SATA'),
        ('SAS','SAS'),
        ('SCSI','SCSI'),
        ('SSD','SSD')
    )
 
    iface_type = models.CharField(u'接口类型',max_length=64,choices=disk_iface_choice,default='SAS')
    memo = models.TextField(u'备注',blank=True,null=True)
    create_date = models.DateTimeField(blank=True, auto_now_add=True)
    update_date = models.DateTimeField(blank=True,null=True,auto_now=True)
 
    auto_create_fields = ['sn','slot','manufactory','model','capacity','iface_type']
    class Meta:
        unique_together = ['asset','slot']
        verbose_name = '硬盘'
        verbose_name_plural = '硬盘'
    def __str__(self):
        return '%s:slot:%s capacity:%s' %(self.asset_id,self.slot,self.capacity)
 

class NIC(models.Model):
    asset = models.ForeignKey('Asset')
    name = models.CharField(u'网卡名',max_length=64,blank=True,null=True)
    sn = models.CharField(u'SN号',max_length=128,blank=True,null=True)
    model = models.CharField(u'网卡型号',max_length=128,blank=True,null=True)
    macaddress = models.CharField(u'MAC',max_length=64,unique=True)
    ipaddress = models.GenericIPAddressField(u'IP',blank=True,null=True)
    netmask = models.CharField(max_length=64,blank=True,null=True)
    bonding = models.CharField(max_length=64,blank=True,null=True)
    memo = models.CharField(u'备注',max_length=128,blank=True,null=True)
    create_date = models.DateTimeField(blank=True, auto_now_add=True)
    update_date = models.DateTimeField(blank=True,null=True,auto_now=True)
 
    def __str__(self):
        return '%s:%s' %(self.asset_id,self.macaddress)
 
    class Meta:
        verbose_name = u'网卡'
        verbose_name_plural = u'网卡'
    auto_create_fields = ['name','sn','model','macaddress','ipaddress','netmask','bonding']
    
    
class RaidAdaptor(models.Model):
    asset = models.ForeignKey('Asset')
    sn = models.CharField(u'SN型号',max_length=128,blank=True,null=True)
    slot = models.CharField(u'插口',max_length=64)
    model = models.CharField(u'型号',max_length=64,blank=True,null=True)
    memo = models.TextField(u'备注',blank=True,null=True)
    create_date = models.DateTimeField(blank=True, auto_now_add=True)
    update_date = models.DateTimeField(blank=True,null=True,auto_now=True)
 
    def __str__(self):
        return self.name
    class Meta:
        unique_together = ('asset','slot')

        
class Manufactory(models.Model):
    manufactory = models.CharField(u'厂商名称',max_length=64,unique=True)
    support_num = models.CharField(u'支持电话',max_length=30,blank=True,null=True)
    memo = models.CharField(u'备注',max_length=128,blank=True)
    
    def __str__(self):
        return self.manufactory
    
    class Meta:
        verbose_name = '厂商'
        verbose_name_plural = '厂商'
 

class BusinessUnit(models.Model):
    parent_unit = models.ForeignKey('self',related_name='parent_level',blank=True,null=True)
    name = models.CharField(u'业务线',max_length=64,unique=True)
    memo = models.CharField(u'备注',max_length=128,blank=True)
 
    def __str__(self):
        return self.name
    
    class Meta:
        verbose_name = '业务线'
        verbose_name_plural = '业务线'
        
        
class Contract(models.Model):
    sn = models.CharField(u'合同号',max_length=128,unique=True)
    name = models.CharField(u'合同名称',max_length=64)
    memo = models.TextField(u'备注',blank=True,null=True)
    price = models.IntegerField(u'合同金额')
    detail = models.TextField(u'合同详细',blank=True,null=True)
    start_date = models.DateField(blank=True)
    end_date = models.DateField(blank=True)
    license_num = models.IntegerField(u'license数量',blank=True,)
    create_date = models.DateTimeField(blank=True, auto_now_add=True)
    update_date = models.DateTimeField(blank=True,null=True,auto_now=True)
    
    class Meta:
        verbose_name = '合同'
        verbose_name_plural = '合同'
        
    def __str__(self):
        return self.name
 

class IDC(models.Model):
    name = models.CharField(u'机房名称',max_length=64,unique=True)
    memo = models.CharField(u'备注',max_length=128,blank=True,null=True)
 
    def __str__(self):
        return self.name
    
    class Meta:
        verbose_name = '机房'
        verbose_name_plural = '机房'
 

class Tag(models.Model):
    name = models.CharField('Tag name',max_length=32,unique=True)
    creater = models.ForeignKey(UserProfile)
    create_date = models.DateField(auto_now_add=True)
    
    def __str__(self):
        return self.name
 

class EventLog(models.Model):
    name = models.CharField(u'事件名称',max_length=128)
    event_type_choices = (
        (1,u'硬件变更'),
        (2,u'新增配件'),
        (3,u'设备下线'),
        (4,u'设备上线'),
        (5,u'定期维护'),
        (6,u'设备上线\更新\变更'),
        (7,u'其他'),
    )
    event_type = models.SmallIntegerField(u'事件类型',choices=event_type_choices)
    asset = models.ForeignKey('Asset')
    component = models.CharField('事件子项',max_length=255,blank=True,null=True)
    detail = models.TextField(u'事件详情')
    date = models.DateTimeField(u'事件时间',auto_now_add=True)
    user = models.ForeignKey(UserProfile,verbose_name=u'事件源')
    memo = models.TextField(u'备注',blank=True,null=True)
 
    def __str__(self):
        return self.name
    
    class Meta:
 
        verbose_name = '事件纪录'
        verbose_name_plural = '事件纪录'
 
    def colored_event_type(self):
        if self.event_type == 1:
            cell_html = '<span style = "background: orange;">%s</span>'
        elif self.event_type == 2:
            cell_html = '<span style="background: yellowgreen;">%s</span>'
        else:
            cell_html = '<span >%s</span>'
        return cell_html % self.get_event_type_display()
    
    colored_event_type.allow_tags = True
    colored_event_type.short_description = u'事件类型'

    
class NewAssetApprovalZone(models.Model):
    sn = models.CharField(u'资产SN号',max_length=128,unique=True)
    asset_type_choices = (
        ('server',u'服务器'),
        ('switch',u'交换机'),
        ('router',u'路由器'),
        ('firewall',u'防火墙'),
        ('storage',u'存储设备'),
        ('NLB',u'NetScaler'),
        ('wireless',u'无线AP'),
        ('software',u'软件资产'),
        ('others',u'其他类'),
    )
    asset_type = models.CharField(choices=asset_type_choices,max_length=64,blank=True,null=True)
    manufactory = models.CharField(max_length=64,blank=True,null=True)
    model = models.CharField(max_length=128,blank=True,null=True)
    ram_size = models.IntegerField(blank=True,null=True)
    cpu_model = models.CharField(max_length=128,blank=True,null=True)
    cpu_count = models.IntegerField(blank=True,null=True)
    cpu_core_count = models.IntegerField(blank=True,null=True)
    os_distribution = models.CharField(max_length=64,blank=True,null=True)
    os_type = models.CharField(max_length=64,blank=True,null=True)
    os_release = models.CharField(max_length=64,blank=True,null=True)
    data = models.TextField(u'资产数据')
    date = models.DateTimeField(u'汇报日期',auto_now_add=True)
    approved = models.BooleanField(u'已批准',default=False)
    approved_by = models.ForeignKey(UserProfile,verbose_name=u'批准人',blank=True,null=True)
    approved_date = models.DateTimeField(u'批准日期',blank=True,null=True)
 
    def __str__(self):
        return self.sn

    class Meta:
        verbose_name = '新上线待批准资产'
        verbose_name_plural = '新上线待批准资产'

逻辑核心-资产自动汇报

视图层demo:

from django.shortcuts import render,HttpResponse
from django.views.decorators.csrf import csrf_exempt
import json
from Sansa import core

# Create your views here.
@csrf_exempt #免除csrf验证
def asset_report(request):
    if request.method == 'POST':
        ass_handler = core.Asset(request)
        if ass_handler.data_is_valid(): #验证数据是否合法
            #注射数据
            ass_handler.data_inject()
        return HttpResponse(json.dumps(ass_handler.response))
    return HttpResponse('----test----')

Core code:

import json
from django.core.exceptions import ObjectDoesNotExist
from Sansa import models
from django.utils import timezone
 
class Asset(object):
    def __init__(self,request):
        self.request = request #view中的request
        #必须要有的字段
        self.manufactory_fields = ['sn','asset_id','asset_type']
        self.field_sets = {
            'asset':['manufactory'],
            'server':['model',
                      'cpu_count',
                      'cpu_core_count',
                      'cpu_model',
                      'raid_type',
                      'os_type',
                      'os_distribution',
                      'os_release'],
            'networkdevice':[]
        }
        #给客户端的返回
        self.response = {
            'error':[],
            'info':[],
            'warning':[],
        }
        
    def response_msg(self,msg_type,key,msg):
        '''
        将要返回的值放入self.response字典中
        :param msg_type: 数据类型 如:error
        :param key:
        :param msg:
        :return:
        '''
        if msg_type in self.response:
            self.response[msg_type].append({key:msg})
        else:
            raise ValueError
            
    def mandatory_check(self,data,only_check_sn=False):
        '''
        强制性检测函数
        :param data:  传入的数据
        :param only_check_sn:
        :return:
        '''
        #遍历强制性检测字段,判断字段有没有
        for field in self.manufactory_fields:
            #没有写入到error中
            if not field in data:
                self.response_msg('error','MandatoryCheckFailed',"The field [%s] is mandatory and not provided in your reporting data" % field)
        #如果有错误直接返回不在往下进行
        else:
            if self.response['error']:return False
        try:
            if not only_check_sn:  #不是新资产
                #获取这条资产的对象
                self.asset_obj = models.Asset.objects.get(id=int(data['asset_id']),sn=data['sn'])
            else: #是新资产
                self.asset_obj = models.Asset.objects.get(sn=data['sn'])
            return True #前置检测通过
        except ObjectDoesNotExist as e:
            self.response_msg('error',
                              'AssetDataInvalid',
                              "Cannot find asset object in DB by using asset id [%s] and SN [%s] " % (data['asset_id'],data['sn']))
            self.waiting_approval = True
            return False
        
    def get_asset_id_by_sn(self):
        data = self.request.POST.get('asset_data')
        response = {}
        if data:
            try:
                data = json.loads(data)
                #这个资产已经在数据库中
                if self.mandatory_check(data,only_check_sn=True):
                    response = {'asset_id':self.asset_obj.id}
                else:
                    if hasattr(self,'waiting_approval'):
                        response = {'needs_approval':"this is a new asset,needs IT admin's approval to create the new asset id."}
                        self.clean_data = data
                        self.save_new_asset_to_approval_zone()
                    else:
                        response = self.response
            except ValueError as e:
                self.response_msg('error','AssetDataInvalid',str(e))
                response = self.response
        else:
            self.response_msg('error','AssetDataInvalid',"The reported asset data is not valid or provided")
            response = self.response
        return response
 
    def save_new_asset_to_approval_zone(self):
        asset_sn = self.clean_data.get('sn')
        asset_already_in_approval_zone = models.NewAssetApprovalZone.objects.get_or_create(
            sn = asset_sn,
            data = json.dumps(self.clean_data),
            manufactory = self.clean_data.get('manufactory'),
            model = self.clean_data.get('model'),
            asset_type = self.clean_data.get('asset_type'),
            ram_size = self.clean_data.get('ram_size'),
            cpu_model = self.clean_data.get('cpu_model'),
            cpu_count = self.clean_data.get('cpu_count'),
            cpu_core_count = self.clean_data.get('cpu_core_count'),
            os_distribution = self.clean_data.get('os_distribution'),
            os_release = self.clean_data.get('os_release'),
            os_type = self.clean_data.get('os_type'),
        )
        return True
    
    def data_is_valid(self):
        '''
        数据检测
        :return:
        '''
        data = self.request.POST.get('asset_data')
        #确保存在数据
        if data:
            try:
                data = json.loads(data)
                #强制性检测
                self.mandatory_check(data)
                self.clean_data = data
                #如果没有错误就往下进行
                if not self.response['error']:
                    return True
            except ValueError as e:
                self.response_msg('error','AssetDataInvalid',str(e))
        else:
            self.response_msg('error','AssetDataInvalid',"The reported asset data is not valid or provided")
            
    def _is_new_asset(self):
        '''
        判断是否是新资产
        :return:
        '''
        #是新资产
        if not hasattr(self.asset_obj,self.clean_data['asset_type']):
            return True
        #不是新资产
        else:
            return False
        
    def data_inject(self):
        '''
        注射
        :return:
        '''
        #判断是否是新资产
        if self._is_new_asset():
            #创建新资产
            self.create_asset()
        else:
            self.update_asset()
            
    def data_is_valid_without_id(self):
        data = self.request.POST.get('asset_data')
        if data:
            try:
                data = json.loads(data)
                asset_obj = models.Asset.objects.get_or_create(
                    sn = data.get('sn'),
                    name = data.get('name')
                )
                data['asset_id'] = asset_obj[0].id
                self.mandatory_check(data)
                self.clean_data = data
                if not self.response['error']:
                    return True
            except ValueError as e:
                self.response_msg('error','AssetDataInvalid',str(e))
        else:
             self.response_msg('error','AssetDataInvalid', "The reported asset data is not valid or provided")
                
    def reformat_components(self,identify_field,data_set):
        for k,data in data_set.items():
            data[identify_field] = k
            
    def _verify_field(self,data_set,field_key,data_type,required=True):
        '''
        判断客户端传来的数据中的field_key的数据类型是不是data_type
        :param data_set: 客户端数据
        :param field_key: 客户端数据中的key
        :param data_type: 数据类型
        :param required:
        :return:
        '''
        field_val = data_set.get(field_key)
        if field_val:
            try:
                data_set[field_key] = data_type(field_val)
            except ValueError as e:
                self.response_msg('error','InvalidField',"The field [%s]'s data type is invalid, the correct data type should be [%s] " % (field_key,data_type))
        elif required == True:
            self.response_msg('error','LackOfField',"The field [%s] has no value provided in your reporting data [%s]" % (field_key,data_set))
    def create_asset(self):
        #创建资产,按资产类型不同调用不同的方法创建,如_create_server
        func = getattr(self,'_create_%s' % self.clean_data['asset_type'])
        create_obj = func()
    def update_asset(self):
        #更新资产,按资产类型不同调用不同的方法更新,如_update_server
        func = getattr(self,'_update_%s' % self.clean_data['asset_type'])
        create_obj = func()
        
    def _update_server(self):
        '''
        服务器更新
        :return:
        '''
        nic = self._update_asset_component(
            data_source = self.clean_data['nic'],
            fk = 'nic_set',
            update_fields = ['name','sn','model','macaddress','ipaddress','netmask','bonding'],
            identify_field = 'macaddress'
        )
        disk = self._update_asset_component(
            data_source = self.clean_data['physical_disk_driver'],
            fk = 'disk_set',
            update_fields = ['slot','sn','model','manufactory','capacity','iface_type'],
            identify_field = 'slot',
        )
        ram = self._update_asset_component(
            data_source = self.clean_data['ram'],
            fk = 'ram_set',
            update_fields = ['slot','sn','model','capacity'],
            identify_field = 'slot',
        )
        cpu = self._update_cpu_component()
        manufactory = self._update_manufactory_component()
        server = self._update_server_component()
        
    def _create_server(self):
        self._create_server_info()
        self._create_or_update_manufactory()
        self._create_cpu_component()
        self._create_disk_component()
        self._create_nic_component()
        self._create_ram_component()
 
        log_msg = "Asset [<a href='/admin/assets/asset/%s' target='_blank'>%s</a>] has been created!" % (self.asset_obj.id,self.asset_obj)
        self.response_msg('info','NewAssetOnline',log_msg)
        
    def _create_server_info(self,ignore_errs=False):
        '''
        创建服务器信息
        :param ignore_errs:忽略错误
        :return:
        '''
        try:
            #判断数据类型
            self._verify_field(self.clean_data,'model',str)
            #如果没有错误或忽略错误
            if not len(self.response['error']) or ignore_errs == True:
                data_set = {
                    'asset_id' : self.asset_obj.id,
                    'raid_type' : self.clean_data.get('raid_type'),
                    'model' : self.clean_data.get('model'),
                    'os_type' : self.clean_data.get('os_type'),
                    'os_distribution' : self.clean_data('os_distribution'),
                    'os_release' : self.clean_data.get('os_release')
                }
                #创建server表
                obj = models.Server(**data_set)
                #保存
                obj.save()
                return obj
        except Exception as e:
            self.response_msg('error','ObjectCreationException','Object [server] %s' % str(e) )
 
    def _create_or_update_manufactory(self,ignore_errs=False):
        '''
        创建或更新厂商
        :param ignore_errs:
        :return:
        '''
        try:
            #确保厂商字段存在并且类型正确
            self._verify_field(self.clean_data,'manufactory',str)
            #获取厂商名字
            manufactory = self.clean_data.get('manufactory')
            #判断是否有错误
            if not len(self.response['error']) or ignore_errs == True:
                #判断数据库中是否有该厂商
                obj_exist = models.Manufactory.objects.filter(manufactory=manufactory)
                #如果有就赋给该server
                if obj_exist:
                    obj = obj_exist[0]
                #没有该厂商就创建厂商
                else:
                    obj = models.Manufactory(manufactory=manufactory)
                    obj.save()
                #将厂商赋给该资产
                self.asset_obj.manufactory = obj
                #保存
                self.asset_obj.save()
        except Exception as e:
            self.response_msg('error','ObjectCreationException','Object [manufactory] %s' % str(e))
            
    def _create_cpu_component(self,ignore_errs=False):
        '''
        创建cpu资产
        :param ignore_errs:
        :return:
        '''
        try:
            #检测一些字段是否符合要求
            self._verify_field(self.clean_data,'model',str)
            self._verify_field(self.clean_data,'cpu_count',int)
            self._verify_field(self.clean_data,'cpu_core_count',int)
            if not len(self.response['error']) or ignore_errs == True:
                data_set = {
                    'asset_id' : self.asset_obj.id,
                    'cpu_model' : self.clean_data.get('cpu_model'),
                    'cpu_count' : self.clean_data.get('cpu_count'),
                    'cpu_core_count' : self.clean_data('cpu_core_count')
                }
                #创建cpu资产
                obj = models.CPU(**data_set)
                obj.save()
                #放入返回结果,没有创建日志
                log_msg = "Asset[%s] --> has added new [cpu] component with data [%s]" %(self.asset_obj,data_set)
                self.response_msg('info','NewComponentAdded',log_msg)
                return obj
        except Exception as e:
            self.response_msg('error','ObjectCreationException','Object [cpu] %s' % str(e))
            
    def _create_disk_component(self):
        '''
        创建硬盘,可以有多块硬盘
        :return:
        '''
        disk_info = self.clean_data.get('physical_disk_driver')
        if disk_info:
            #遍历硬盘信息
            for disk_item in disk_info:
                try:
                    #信息检测
                    self._verify_field(disk_item,'slot',str)
                    self._verify_field(disk_item,'capacity',float)
                    self._verify_field(disk_item,'iface_type',str)
                    self._verify_field(disk_item,'model',str)
                    #是否有错误
                    if not len(self.response['error']):
                        data_set = {
                            'asset_id' : self.asset_obj.id,
                            'sn' : disk_item.get('sn'),
                            'slot' : disk_item.get('slot'),
                            'capacity' : disk_item.get('capacity'),
                            'model' : disk_item.get('model'),
                            'iface_type' : disk_item.get('iface_type'),
                            'manufactory' : disk_item.get('manufactory'),
                        }
                        #写入数据库
                        obj = models.Disk(**data_set)
                        obj.save()
                except Exception as e:
 
                    self.response_msg('error','ObjectCreationException','Object [disk] %s' % str(e))
        else:
            self.response_msg('error','LacOfData','Disk info is not provied in your reporting data')
            
    def _create_nic_component(self):
        '''
        创建网卡,同硬盘
        :return:
        '''
        nic_info = self.clean_data.get('nic')
        if nic_info:
            for nic_item in nic_info:
                try:
                    self._verify_field(nic_item,'macaddress',str)
                    if not len(self.response['error']):
                        data_set = {
                            'asset_id' : self.asset_obj.id,
                            'name' : nic_item.get('name'),
                            'sn' : nic_item.get('sn'),
                            'macaddress' : nic_item.get('macaddress'),
                            'ipaddress' : nic_item.get('ipaddress'),
                            'bonding' : nic_item.get('bonding'),
                            'model' : nic_item.get('model'),
                            'netmask' : nic_item.get('netmask'),
                        }
                        obj = models.NIC(**data_set)
                        obj.save()
                except Exception as e :
                    self.response_msg('error','ObjectCreationException','Object [nic] %s' % str(e) )
        else:
            self.response_msg('error','LackOfData','NIC info is not provied in your reporting data' )
 
    def _create_ram_component(self):
        '''
        创建内存,同硬盘
        :return:
        '''
        ram_info = self.clean_data.get('ram')
        if ram_info:
            for ram_item in ram_info:
                try:
                    self._verify_field(ram_item,'capacity',int)
                    if not len(self.response['error']):
                        data_set = {
                            'asset_id' : self.asset_obj.id,
                            'slot' : ram_item.get('slot'),
                            'sn' : ram_item.get('sn'),
                            'capacity' : ram_item.get('capacity'),
                            'model' : ram_item.get('model'),
                        }
                        obj = models.RAM(**data_set)
                        obj.save()
                except Exception as e:
                    self.response_msg('error','ObjectCreationException','Object [ram] %s' % str(e))
        else:
            self.response_msg('error','LackOfData','RAM info is not provied in your reporting data')
            
    def _update_server_component(self):
        update_fields = ['model','raid_type','os_type','os_distribution','os_release']
        if hasattr(self.asset_obj,'server'):
            self._compare_componet(
                model_obj = self.asset_obj.server,
                fields_from_db = update_fields,
                data_source = self.clean_data,
            )
        else:
            self._create_server_info(ignore_errs=True)
            
    def _update_manufactory_component(self):
        self._create_or_update_manufactory(ignore_errs=True)
        
    def _update_cpu_component(self):
        update_fields = ['cpu_model','cpu_count','cpu_core_count']
        if hasattr(self.asset_obj,'cpu'):
            self._compare_componet(
                model_obj = self.asset_obj.cpu,
                fields_from_db = update_fields,
                data_source = self.clean_data,
            )
        else:
            self._create_cpu_component(ignore_errs=True)
 
    def _update_asset_component(self,data_source,fk,update_fields,identify_field=None):
        '''
        通用的更新做对比的方法
        :param data_source: 客户端数据源,网卡列表之类的
        :param fk:  告诉函数比什么类型资产
        :param update_fields: 比什么字段
        :param identify_field: 在数据库中用什么字段去查询
        :return:
        '''
        try:
            #取到资产的关联字段,类似于models.Asset.objects.all()[0].nic_set
            component_obj = getattr(self.asset_obj,fk)
            if hasattr(component_obj,'select_related'):
                #取到资产的反向关联对象,如models.Asset.objects.all()[0].nic_set.select_related
                objects_from_db = component_obj.select_related()
                #遍历取到的列表对象
                for obj in objects_from_db:
                    #取到数据库中的唯一值,如mac地址的值
                    key_field_data = getattr(obj,identify_field)
                    # if type(data_source) is list:
                    #遍历客户端传过来的数据列表
                    for source_data_item in data_source:
                        #找到客户端传过来的数据中identify_field对应的值,如mac地址
                        key_field_data_from_source_data = source_data_item.get(identify_field)
                        #如果上述值存在
                        if key_field_data_from_source_data:
                            #如果数据库中的唯一值和客户端的相等代表匹配上了,应该比对数据库中的值和客户端的值是否一致
                            if key_field_data == key_field_data_from_source_data:
                                self._compare_componet(
                                    model_obj = obj,
                                    fields_from_db = update_fields,
                                    data_source = source_data_item,
                                )
                                break
                        else:
                            self.response_msg('warning','AssetUpdateWarning',"Asset component [%s]'s key field [%s] is not provided in reporting data " % (fk,identify_field))
                    else:
                        self.response_msg('warning','AssetUpdateWarning',"Cannot find any matches in source data by using key field val [%s],component data is missing in reporting data!" %(key_field_data))
                    # elif type(data_source) is dict:
                    #     for key,source_data_item in data_source.items():
                    #         key_field_data_from_source_data = source_data_item.get(identify_field)
                    #         if key_field_data_from_source_data:
                    #             if key_field_data == key_field_data_from_source_data:
                    #                 self._compare_componet(
                    #                     model_obj = obj,
                    #                     fields_from_db=update_fields,
                    #                     data_source = source_data_item,
                    #                 )
                    #                 break
                    #         else:
                    #             self.response_msg('warning','AssetUpdateWarning',"Asset component [%s]'s key field [%s] is not provided in reporting data " % (fk,identify_field))
                    #     else:
                    #         print('\033[33;1mWarning:cannot find any matches in source data by using key field val [%s],component data is missing in reporting data!\033[0m' %(key_field_data))
                    # else:
                    #     print('\033[31;1mMust be sth wrong,logic should not  goes to here at all.\033[0m')
                #添加或删除的资产
                self._filter_add_or_deleted_components(
                    #表名,如:NIC
                    model_obj_name=component_obj.model._meta.object_name,
                    #数据库中的数据
                    data_from_db=objects_from_db,
                    #客户端数据
                    data_source=data_source,
                    #关键key
                    identify_field=identify_field,
                )
            else:
                pass
        except ValueError as e:
            print('\033[41;1m%s\033[0m' % str(e))
            
    def _filter_add_or_deleted_components(self,model_obj_name,data_from_db,data_source,identify_field):
        #客户端所有的数据列表,如mac地址列表
        data_source_key_list = []
        #遍历客户端数据列表
        for data in data_source:
            #将客户端的数据添加进客户端数据列表中,如网卡中的mac地址
            data_source_key_list.append(data.get(identify_field))
        #将客户端的数据列表变成集合
        data_source_key_list = set(data_source_key_list)
        #将数据库中的数据列表变成集合,如mac地址集合
        data_identify_val_from_db = set([getattr(obj,identify_field) for obj in data_from_db])
        #只在数据库中的数据
        data_only_in_db = data_identify_val_from_db-data_source_key_list
        #只在客户端中的数据
        data_only_in_data_source = data_source_key_list - data_identify_val_from_db
        #只在数据库中的数据应该删除
        self._delete_components(all_components = data_from_db,delete_list = data_only_in_db, identify_field=identify_field)
        #只在数据库中的数据如果存在,应该添加
        if data_only_in_data_source:
            self._add_components(model_obj_name=model_obj_name,all_components=data_source, add_list = data_only_in_data_source, identify_field=identify_field)
            
    def _add_components(self,model_obj_name,all_components,add_list,identify_field):
        '''
        添加资产记录
        :param model_obj_name: 数据库表名,如NIC
        :param all_components: 所有的资产对象,客户端的资产对象,如nic_obj1,如nic_obj2
        :param add_list: 添加列表
        :param identify_field:
        :return:
        '''
        #获取表结构的类,如models.NIC
        model_class = getattr(models,model_obj_name)
        #将要添加数据列表,如[mac1,mac2]
        will_be_creating_list = []
        #遍历所有的资产
        for data in all_components:
            #判断资产在不在添加列表中
            if data[identify_field] in add_list:
                #在添加列表中就放入添加数据列表
                will_be_creating_list.append(data)
        try:
            #遍历在添加列表中就放入添加数据列表
            for component in will_be_creating_list:
                #创建数据的字典
                data_set = {}
                #要创建的字段,每一个表时不一样的
                for field in model_class.auto_create_fields:
                    #将客户端的数据写入data_set
                    data_set[field] = component.get(field)
                #写入到哪个资产中
                data_set['asset_id'] = self.asset_obj.id
                #写入数据库
                obj = model_class(**data_set)
                obj.save()
                log_msg = "Asset[%s] --> component[%s] has justed added a new item [%s]" %(self.asset_obj,model_obj_name,data_set)
                self.response_msg('info','NewComponentAdded',log_msg)
                #写入日志
                log_handler(self.asset_obj,'NewComponentAdded',self.request.user,log_msg,model_obj_name)
        except Exception as e:
            log_msg = "Asset[%s] --> component[%s] has error: %s" %(self.asset_obj,model_obj_name,str(e))
            self.response_msg('error',"AddingComponentException",log_msg)
            
    def _delete_components(self,all_components,delete_list,identify_field):
        '''
        删除资产函数
        :param all_components: 所有的资产对象,数据库的资产对象,如nic_obj1,如nic_obj2
        :param delete_list: 要删除的唯一数据列表
        :param identify_field:
        :return:
        '''
        #要删除的资产对象列表
        deleting_obj_list = []
        for obj in all_components:
            val = getattr(obj,identify_field) #取出数据库中的值,如mac地址
            if val in delete_list:  #如[mac1,mac2]
                deleting_obj_list.append(obj) #添加的删除对象列表中
        #遍历删除列表并删除
        for i in deleting_obj_list:
            log_msg = "Asset[%s] --> component[%s] --> is lacking from reporting source data, assume it has been removed or replaced,will also delete it from DB" %(self.asset_obj,i)
            self.response_msg('info','HardwareChanges',log_msg)
            #添加到日志中
            log_handler(self.asset_obj,'HardwareChanges',self.request.user,log_msg,i)
            i.delete()
            
    def _compare_componet(self,model_obj,fields_from_db,data_source):
        '''
        比较客户端字段和数据库的字段
        :param model_obj: 数据库中的一个对象,如nic的对象
        :param fields_from_db: 数据库中要比较的字段
        :param data_source:  客户端具体的一块网卡的数据,如一块网卡的数据
        :return:
        '''
        #遍历要检测的字段
        for field in fields_from_db:
            #数据库中的数据
            val_from_db = getattr(model_obj,field)
            #取出客户端的数据
            val_from_data_source = data_source.get(field)
            #确认客户端数据存在
            if val_from_data_source:
                #转换数据格式,和数据库中的一致
                if type(val_from_db) in (int,):val_from_data_source = int(val_from_data_source)
                elif type(val_from_db) is float:val_from_data_source = float(val_from_data_source)
                #相等不做任何变化
                if val_from_db == val_from_data_source:
                    pass
                else:
                    #获取字段对象
                    db_field = model_obj._meta.get_field(field)
                    #修改数据
                    db_field.save_form_data(model_obj,val_from_data_source)
                    model_obj.update_date = timezone.now()
                    #保存
                    model_obj.save()
                    log_msg = "Asset[%s] --> component[%s] --> field[%s] has changed from [%s] to [%s]" %(self.asset_obj,model_obj,field,val_from_db,val_from_data_source)
                    self.response_msg('info','FieldChanged',log_msg)
                    #日志记录
                    log_handler(self.asset_obj,'FieldChanged',self.request.user,log_msg,model_obj)
            else:
                self.response_msg('warning','AssetUpdateWarning',"Asset component [%s]'s field [%s] is not provided in reporting data " % (model_obj,field))
        model_obj.save()
        
def log_handler(asset_obj,event_name,user,detail,component=None):
    log_catelog = {
        1 : ['FieldChanged','HardwareChanges'],
        2 : ['NewComponentAdded'],
    }
    if not user.id:
        user = models.UserProfile.objects.filter(is_admin=True).last()
    event_type = None
    for k,v in log_catelog.items():
        if event_name in v:
            event_type = k
            break
    log_obj = models.EventLog(
        name = event_name,
        event_type = event_type,
        asset_id = asset_obj.id,
        component = component,
        detail = detail,
        user_id = user.id
    )
    log_obj.save()

(完)

PREVIOUSGo语言编程环境搭建