When building data intensive applications, a significant portion of your time will be dedicated to data wrangling (cleaning, de-duping, removing markups, etc.). Data Prep Kit (DPK) is an open source python library that can scale from your laptop to a highly scalable cluster in the cloud. It has been used at scale to prepare terabytes of data to train the IBM Granite Large Language Models (LLMS). Features include:
- de-duping documents (exact dedupe and fuzzy dedupe)
- handling documents and code
- language detection (spoken languages and programming languages)
- removing PII
- malware detection
- creating embeddings for a vector database
Check out the IBM TechXchange Dev Day: Virtual Agents spotlight on IBM Developer: https://ibm.biz/virtual-agents-dev-day-highlights
Connect with developers and other practitioners in the IBM TechXchange Community: https://ibm.biz/techxchange-community
Choose a specific community topic group:
IBM Granite AI foundation models: https://ibm.biz/granite-models-group
Global AI and Data Science: https://ibm.biz/global-ai-data-science-group
watsonx.ai: https://ibm.biz/watsonx-ai-group
watsonx Assistant: https://ibm.biz/watsonx-assistant-group
watsonx Orchestrate: https://ibm.biz/watsonx-orchestrate-group
____________________________________________
IBM Developer — write better code, boost your skills, and build something new: https://ibm.biz/ibm-developer-yt
Subscribe to see more developer content: https://ibm.biz/ibm-developer-yt-subscribe
Follow IBM Developer on LinkedIn: https://ibm.biz/ibm-developer-linkedin-yt
More from IBM Developer:
Community: https://community.ibm.com/community/user/community
Blog: https://developer.ibm.com/blogs/
Call for Code: https://developer.ibm.com/callforcode/
#virtualagents
#IBMTechXchange
#IBMDeveloper
#Developer
#Coding