腾讯开放TDinsight机器学习平台等政企大数据平台

责任编辑:editor004

作者:陈利鑫

2017-12-18 11:18:25

摘自:INFOQ

2017 年 6 月 16 日,腾讯新一代高性能计算平台 Angel 在 Github 上低调开源。

2017 年 6 月 16 日,腾讯新一代高性能计算平台 Angel 在 Github 上低调开源。时隔半年,12 月 13 日,腾讯在“2017互联网+大数据高峰论坛”发布“腾讯慧聚”品牌,其中就包括机器学习基础平台TDinsight。与Angel和其他机器学习平台相比,TDinsight有何优势?

TDinsight机器学习平台

“腾讯慧聚”包括五大数据平台,分别是大数据一站式平台Dmaster、大规模事务处理平台Tbase、大数据实时接入平台TDbank、大数据实时多维分析平台Hermes,以及机器学习基础平台TDinsight。

据腾讯互联网+大数据产品中心总经理刘煜宏介绍,TDinsight机器学习平台提供一站式的机器学习平台,通过可视化的拖曳布局,组合各种数据源、组件、算法、模型和评估模块,支持各种主流的开源机器学习框架,包括Spark、Python、R、XGBoost。覆盖特征工程、分类、聚类、回归、关联规则、时间序列等传统机器学习算法的同时,支持图算法、深度学习等更加丰富的算法库,让用户可以快速接入人工智能,释放数据潜力。

那么,TDinsight机器学习平台相比其他相似产品有何优势?这个平台是否开源?是否意味着腾讯以后将会开放自己的AI能力呢?

对此,腾讯互联网+大数据产品中心总经理刘煜宏说道:“腾讯有几个AI部门,包括提到的优图、医疗觅影,就是很好的AI跟行业结合很好的案例,所以腾讯AI能力一直体现在我们产品里,现在也单独拿出来开放了。TDinsight是机器学习基础平台,腾讯大数据去年发布的Angel在6月份开源了,Angel是一个面向机器学习的分布式高性能计算平台。那Angel跟TDinsight是什么关系呢?其实TDinsight你可以认为是一个机器学习的调度平台,但是又不仅仅是调度平台,TDinsight自身包含多种算法以及模型,并且支持多源的输入以及输出,TDinsight采用拖拽的方式能够根据不同的算法、模型调度对应不同的机器学习组件(框架),例如:Angel、Spark、TensorFlow、Torch等,完成机器学习整个流程。”

虽然TDinsight目前已经对政企开放,但开源似乎还是一件遥不可期的事情,刘煜宏表示,“我们也是跟各行各业的定制需求结合,目前要开源出来还不是很好的时机,现在腾讯公司开源的也越来越多,包括大数据是来源于开源。我们还是会回归到社区里,包括Tbase,已经与社区结合得非常紧密,是非常核心的开源,包括资源调度管理平台,调度是在全球计算能力领先的很重要的模块。所以大数据开源会越来越多,但不像安卓整体开源,我们也会结合社区化把很多东西反馈到里面。”

Angel机器学习平台

Angel平台是使用Java和Scala混合开发的机器学习框架,用户可以像用Spark, MapReduce一样,用它来完成机器学习的模型训练。2017 年 6 月 16 日,腾讯新一代高性能计算平台 Angel 在 Github 上低调开源。

Angel采用参数服务器架构,支持十亿级别维度的模型训练。采用了多种业界最新技术和腾讯自主研发技术,如SSP(Stale synchronous Parallel)、异步分布式SGD、多线程参数共享模式HogWild、网络带宽流量调度算法、计算和网络请求流水化、参数更新索引和训练数据预处理方案等。

这些技术使Angel性能大幅提高,达到常见开源系统Spark的数倍到数十倍,能在千万到十亿级的特征维度条件下运行。

自2016年初在腾讯内部上线以来,Angel已应用于腾讯视频、腾讯社交广告及用户画像挖掘等精准推荐业务。未来还将不断拓展应用场景,目标是支持腾讯等企业级大规模机器学习任务。

Angel相关链接:https://s.geekbang.org/search/c=0/k=Angel/t=

感谢徐川对本文的审校。

给InfoQ中文站投稿或者参与内容翻译工作,请邮件至editors@cn.infoq.com。也欢迎大家通过新浪微博(@InfoQ,@丁晓昀),微信(微信号:InfoQChina)关注我们。

评价本文

专业度风格编辑观点主编观点              此内容所在的主题为语言 & 开发告诉我们您的想法社区评论         2860000},{"score":77901,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/presentations/tencent-pigeon-real-time-accurate-push-system/zh/smallimage/ganhengtong100-1512916635657.jpg","url":"http://www.infoq.com/cn/presentations/tencent-pigeon-real-time-accurate-push-system","title":"腾讯信鸽实时精准推送系统的演进与实践","authorsList":["甘恒通"],"itemPath":"/presentations/tencent-pigeon-real-time-accurate-push-system","contentType":"presentations","date":1512945360000},{"score":75815,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/presentations/big-data-platforms-and-architectures-for-global-services/zh/smallimage/renxiawei100-1512039472271.jpg","url":"http://www.infoq.com/cn/presentations/big-data-platforms-and-architectures-for-global-services","title":"精益创新:从 0 到 1 构建服务全球的大数据平台和架构","authorsList":["夏卫"],"itemPath":"/presentations/big-data-platforms-and-architectures-for-global-services","contentType":"presentations","date":1512340740000},{"score":68001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/build-deploy-scalable-machine-learning-production-kafka/zh/smallimage/GettyImages-578583314-2-1510071286006.jpg","url":"http://www.infoq.com/cn/articles/build-deploy-scalable-machine-learning-production-kafka","title":"在生产环境使用Kafka构建和部署大规模机器学习","authorsList":["Kai Waehner"],"itemPath":"/articles/build-deploy-scalable-machine-learning-production-kafka","contentType":"articles","date":1510096200000},{"score":67413,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/yiguan-bigdata-architecture-evolution-guowei/zh/smallimage/api-facades-logo-1509891971352.jpg","url":"http://www.infoq.com/cn/articles/yiguan-bigdata-architecture-evolution-guowei","title":"易观 CTO 郭炜:易观大数据架构的变迁","authorsList":["赵新龙"],"itemPath":"/articles/yiguan-bigdata-architecture-evolution-guowei","contentType":"articles","date":1509923940000},{"score":67413,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/big-data-in-finance/zh/smallimage/logo-ieee-1508304511929.jpeg","url":"http://www.infoq.com/cn/articles/big-data-in-finance","title":"海量数据与海量金钱:大数据在金融领域的作用","authorsList":["Jennifer Q. Trelewicz"],"itemPath":"/articles/big-data-in-finance","contentType":"articles","date":1509923700000},{"score":65901,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/prepare-text-data-machine-learning-scikit-learn/zh/smallimage/GettyImages-508140566-1509378737592.jpg","url":"http://www.infoq.com/cn/articles/prepare-text-data-machine-learning-scikit-learn","title":"如何使用Scikit-learn实现用于机器学习的文本数据准备","authorsList":["Jason Brownlee"],"itemPath":"/articles/prepare-text-data-machine-learning-scikit-learn","contentType":"articles","date":1509490080000},{"score":63512,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/industrial-big-data/zh/smallimage/redis-logo-1508773153565.jpg","url":"http://www.infoq.com/cn/articles/industrial-big-data","title":"三位一体的工业大数据综述","authorsList":["朱武"],"itemPath":"/articles/industrial-big-data","contentType":"articles","date":1508797920000},{"score":61413,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/self-discipline-of-machine-learning-platform/zh/smallimage/logo-mobile-1508167959202.jpeg","url":"http://www.infoq.com/cn/articles/self-discipline-of-machine-learning-platform","title":"道器相融,论一个优秀机器学习平台的自我修养","authorsList":["黄明"],"itemPath":"/articles/self-discipline-of-machine-learning-platform","contentType":"articles","date":1508195640000},{"score":61401,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/machine-learning-in-blockchain-technologies/zh/smallimage/logo232-1508062753415.jpeg","url":"http://www.infoq.com/cn/articles/machine-learning-in-blockchain-technologies","title":"区块链技术中的机器学习","authorsList":["Cryptics"],"itemPath":"/articles/machine-learning-in-blockchain-technologies","contentType":"articles","date":1508193120000},{"score":59001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/four-dimensions-of-large-machine-learning-framework/zh/smallimage/budong.jpg","url":"http://www.infoq.com/cn/articles/four-dimensions-of-large-machine-learning-framework","title":"后台程序员转算法的参考秘籍:大规模机器学习框架的四重境界","authorsList":["张红林"],"itemPath":"/articles/four-dimensions-of-large-machine-learning-framework","contentType":"articles","date":1507501560000},{"score":57801,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/a-comparison-of-distributed-machine-learning-platforms/zh/smallimage/series-logo-small-1506775223348.jpg","url":"http://www.infoq.com/cn/articles/a-comparison-of-distributed-machine-learning-platforms","title":"分布式机器学习平台大比拼:Spark、PMLS、TensorFlow、MXNet","authorsList":["Murat Demirbas"],"itemPath":"/articles/a-comparison-of-distributed-machine-learning-platforms","contentType":"articles","date":1507161120000},{"score":50010,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/from-distributed-management-to-multi-tenant-implementation/zh/smallimage/logo-2 (1).jpg","url":"http://www.infoq.com/cn/articles/from-distributed-management-to-multi-tenant-implementation","title":"从分布式管理到多租户实现,企业级大数据系统如何利用开源生态构建?","authorsList":["陈冬"],"itemPath":"/articles/from-distributed-management-to-multi-tenant-implementation","contentType":"articles","date":1503439320000},{"score":50010,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/qiniu-big-data-platform-evolution-and-analysis/zh/smallimage/logo-ieee.jpg","url":"http://www.infoq.com/cn/articles/qiniu-big-data-platform-evolution-and-analysis","title":"七牛大数据平台的演进与大数据分析实践","authorsList":["孙健波"],"itemPath":"/articles/qiniu-big-data-platform-evolution-and-analysis","contentType":"articles","date":1503008400000},{"score":50010,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/presentations/kuaishou-live-experience-optimization/zh/smallimage/luowei100.jpg","url":"http://www.infoq.com/cn/presentations/kuaishou-live-experience-optimization","title":"快手在大数据驱动下的直播体验优化","authorsList":["罗喆"],"itemPath":"/presentations/kuaishou-live-experience-optimization","contentType":"presentations","date":1501455000000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/data-virtualization--ai-and-machine-learning/zh/smallimage/agile-logo.jpg","url":"http://www.infoq.com/cn/articles/data-virtualization--ai-and-machine-learning","title":"数据虚拟化:为AI与机器学习实现数据解锁","authorsList":["Robert Alexander"],"itemPath":"/articles/data-virtualization--ai-and-machine-learning","contentType":"articles","date":1504758060000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/cortana-azure-machine-learning/zh/smallimage/agile.jpg","url":"http://www.infoq.com/cn/articles/cortana-azure-machine-learning","title":"Cortana智能与机器学习博客 将人工智能引入商务智能——Azure Machine Learning中的文本分析","authorsList":["Mary Wahl"],"itemPath":"/articles/cortana-azure-machine-learning","contentType":"articles","date":1504736160000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/summary-of-benchmark-algorithms-for-fast-machine-learning/zh/smallimage/agile-logo (1).jpg","url":"http://www.infoq.com/cn/articles/summary-of-benchmark-algorithms-for-fast-machine-learning","title":"总结自快速机器学习算法基准测试的重要经验","authorsList":["Miguel Fierro"],"itemPath":"/articles/summary-of-benchmark-algorithms-for-fast-machine-learning","contentType":"articles","date":1504733400000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/program-techniques-computational-models-xgboost-mxnet/zh/smallimage/logo 112 (3).jpg","url":"http://www.infoq.com/cn/articles/program-techniques-computational-models-xgboost-mxnet","title":"大规模机器学习的编程技术、计算模型以及Xgboost和MXNet案例","authorsList":["陈华清"],"itemPath":"/articles/program-techniques-computational-models-xgboost-mxnet","contentType":"articles","date":1503613080000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/miniflow-build-machine-learning-infrastructure-platform/zh/smallimage/agile-logo.jpg","url":"http://www.infoq.com/cn/articles/miniflow-build-machine-learning-infrastructure-platform","title":"从算法实现到MiniFlow实现,打造机器学习的基础架构平台","authorsList":["陈迪豪"],"itemPath":"/articles/miniflow-build-machine-learning-infrastructure-platform","contentType":"articles","date":1501540920000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/agile-development-of-artificial-intelligence-applications/zh/smallimage/java-logo2.jpg","url":"http://www.infoq.com/cn/articles/agile-development-of-artificial-intelligence-applications","title":"机器学习的最小可用产品:人工智能应用的敏捷开发","authorsList":["田枫"],"itemPath":"/articles/agile-development-of-artificial-intelligence-applications","contentType":"articles","date":1501193820000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/presentations/enter-tencent-audio-and-video-quality-system/zh/smallimage/luobida100.jpg","url":"http://www.infoq.com/cn/presentations/enter-tencent-audio-and-video-quality-system","title":"直面音视频质量评估之痛——走进腾讯音视频质量体系","authorsList":["罗必达"],"itemPath":"/presentations/enter-tencent-audio-and-video-quality-system","contentType":"presentations","date":1500850500000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/machine-learning-with-javascript-part02/zh/smallimage/logo (23).jpg","url":"http://www.infoq.com/cn/articles/machine-learning-with-javascript-part02","title":"机器学习与JavaScript(二)","authorsList":["Abhishek Soni"],"itemPath":"/articles/machine-learning-with-javascript-part02","contentType":"articles","date":1499379540000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/why-tencent-embrace-open-source/zh/smallimage/agile-logo (1).jpg","url":"http://www.infoq.com/cn/articles/why-tencent-embrace-open-source","title":"一向封闭的腾讯,为什么也开始拥抱开源了?","authorsList":["郭蕾"],"itemPath":"/articles/why-tencent-embrace-open-source","contentType":"articles","date":1499293140000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/machine-learning-with-javascript-part01/zh/smallimage/logo (23).jpg","url":"http://www.infoq.com/cn/articles/machine-learning-with-javascript-part01","title":"机器学习与JavaScript(一)","authorsList":["Abhishek Soni"],"itemPath":"/articles/machine-learning-with-javascript-part01","contentType":"articles","date":1499120280000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/ios-11-machine-learning-for-everyone/zh/smallimage/logo-devops.jpg","url":"http://www.infoq.com/cn/articles/ios-11-machine-learning-for-everyone","title":"iOS 11:人人可体验的机器学习","authorsList":["Matthijs Hollemans"],"itemPath":"/articles/ios-11-machine-learning-for-everyone","contentType":"articles","date":1499033460000},{"score":50001,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/interviews/interview-with-shaoyingxia-talk-machine-learning-practice/zh/smallimage/shaoyingxia100.jpg","url":"http://www.infoq.com/cn/interviews/interview-with-shaoyingxia-talk-machine-learning-practice","title":"专访明略数据邵蓥侠:传统公安领域的机器学习实践","authorsList":["邵蓥侠"],"itemPath":"/interviews/interview-with-shaoyingxia-talk-machine-learning-practice","contentType":"interviews","date":1498515480000},{"score":19,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/minibooks/AI-front-201711/zh/smallimage/100-1512039046816.jpg","url":"http://www.infoq.com/cn/minibooks/AI-front-201711","title":"AI前线(2017年11月)","authorsList":["InfoQ中文站"],"itemPath":"/minibooks/AI-front-201711","contentType":"minibooks","date":1512038700000},{"score":18,"topicsIds":null,"imageStoragePath":null,"url":"http://www.infoq.com/cn/news/2017/11/why-amazon-sagemaker-important","title":"为什么你应该关注Amazon SageMaker","authorsList":["杨赛"],"itemPath":"/news/2017/11/why-amazon-sagemaker-important","contentType":"news","date":1512023400000},{"score":18,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/interviews/interview-with-gurinder-talk-paypal-change-industry-with-data/zh/smallimage/gre100-1511260121674.jpg","url":"http://www.infoq.com/cn/interviews/interview-with-gurinder-talk-paypal-change-industry-with-data","title":"PayPal首席架构师Gurinder:我们正在用数据改变行业","authorsList":["Gurinder"],"itemPath":"/interviews/interview-with-gurinder-talk-paypal-change-industry-with-data","contentType":"interviews","date":1511389260000},{"score":18,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/minibooks/tensorflow-programing/zh/smallimage/100-1510627875939.jpg","url":"http://www.infoq.com/cn/minibooks/tensorflow-programing","title":"深度学习利器:TensorFlow程序设计","authorsList":["武维"],"itemPath":"/minibooks/tensorflow-programing","contentType":"minibooks","date":1510704000000},{"score":17,"topicsIds":null,"imageStoragePath":null,"url":"http://www.infoq.com/cn/news/2017/12/tensorflow-lite","title":"TensorFlow Lite支持设备内置会话建模","authorsList":["Srini Penchikala"],"itemPath":"/news/2017/12/tensorflow-lite","contentType":"news","date":1512604800000},{"score":15,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/fpga-computational-performance/zh/smallimage/logo-small-1510653556493.jpg","url":"http://www.infoq.com/cn/articles/fpga-computational-performance","title":"FPGA掌控计算性能","authorsList":["Rob Taylor"],"itemPath":"/articles/fpga-computational-performance","contentType":"articles","date":1512341400000},{"score":15,"topicsIds":null,"imageStoragePath":null,"url":"http://www.infoq.com/cn/news/2017/11/knowledge-graph-articles","title":"你不得不看的六篇知识图谱落地好文","authorsList":["杜小芳","陈思"],"itemPath":"/news/2017/11/knowledge-graph-articles","contentType":"news","date":1511221200000},{"score":15,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/minibooks/4-Paradigm-Technology/zh/smallimage/100-1508808795050.jpg","url":"http://www.infoq.com/cn/minibooks/4-Paradigm-Technology","title":"架构师特刊:范式大学","authorsList":["InfoQ中文站"],"itemPath":"/minibooks/4-Paradigm-Technology","contentType":"minibooks","date":1508808480000},{"score":14,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/articles/integrate-data-analysis-platform/zh/smallimage/kafak-1511695415195.jpg","url":"http://www.infoq.com/cn/articles/integrate-data-analysis-platform","title":"如何整合复杂技术打造数据分析平台?","authorsList":["万晓川"],"itemPath":"/articles/integrate-data-analysis-platform","contentType":"articles","date":1511910420000},{"score":14,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/minibooks/TensorFlow-indepth/zh/smallimage/100-1506507611643.jpg","url":"http://www.infoq.com/cn/minibooks/TensorFlow-indepth","title":"架构师特刊:深入浅出TensorFlow","authorsList":["郑泽宇"],"itemPath":"/minibooks/TensorFlow-indepth","contentType":"minibooks","date":1508370720000},{"score":14,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/minibooks/toptech10/zh/smallimage/100-1508127572572.jpg","url":"http://www.infoq.com/cn/minibooks/toptech10","title":"中国顶尖技术团队访谈录·第十季","authorsList":["InfoQ中文站"],"itemPath":"/minibooks/toptech10","contentType":"minibooks","date":1508195700000},{"score":13,"topicsIds":null,"imageStoragePath":"https://res.infoq.com/presentations/unit-language-understanding-and-interaction-technology/zh/smallimage/sunke100-1509711991338.jpg","url":"http://www.infoq.com/cn/presentations/unit-language-understanding-and-interaction-technology","title":"UNIT:语言理解与交互技术","authorsList":["孙珂"],"itemPath":"/presentations/unit-language-understanding-and-interaction-technology","contentType":"presentations","date":1510182120000}]";var whitepaperVcrsJson = null;var topicSponsorshipJson = "{"iconLink":"/infoq/url.action?i=17062&t=f","iconHref":"https://res.infoq.com/sponsorship/featuredcategory/6821/logo-1512699527324.jpg","id":1580}";var vcrOptionalListJson = null;var contentDatetimeFormat='yyyy年M月d日';var contentUriMapping="news";JSi18n.relatedRightbar_relatedContent='相关内容';JSi18n.relatedRightbar_sponsoredContent='赞助商内容';JSi18n.relatedRightbar_sponsoredBy='赞助商';var topicIds = "2088,2178,171,3169";var communityIds = "2497,2498";var company = ""; var intervalRightbar = setInterval(function() { if (window.vcrsLoaded) { clearInterval(intervalRightbar); if(company != null && company != "") { whitepaperVcrsJson = VCR.filterByCompany(company, window.vcrList); } else { whitepaperVcrsJson = VCR.getByTopicsAndCommunities(window.vcrList, topicIds, communityIds, 5, false, null); } vcrOptionalListJson = VCR.getByTopicsAndCommunities(window.vcrList, topicIds, communityIds, 10, true, null); relatedRightbar.rightbarDisplay(recomJson, whitepaperVcrsJson, topicSponsorshipJson); // track the impression // f_vcrrightbar_sponsorship is available at document ready Tracker.doTrackVcrRightbarImpressions("f_vcrrightbar_sponsorship_top_2"); Tracker.doTrackVcrRightbarBoxesImpressions("f_sponsorbox_top_2"); // only do the tracking here so that all GA vars are initialized if(relatedRightbar.whitepaperWidgetDisplayed){ _gaq.push(['_setCustomVar', 1, 'Whitepaper widget Related Rightbar', "Display", 3]); } optionalVcrBox.parseVendorContentOptionalList(vcrOptionalListJson); // get the version to display optionalVcrBox.abTestVersion = ABTesting.getABTestVersion(); // do the display after page is ready, GA banners all load after page is ready, all GA tracking js vars are available at that time also. No need to do this earlier optionalVcrBox.optionalVcrBoxDisplayAdBlock(); // tracking is done only if not done already! (when the gam event fires before document ready and we do not have the _gaq var available) optionalVcrBox.doTracking("top"); optionalVcrBox.doTracking("bottom"); window.finishedRightbarVcr = true; } }, 200);

链接已复制,快去分享吧

企业网版权所有©2010-2025 京ICP备09108050号-6京公网安备 11010502049343号