Deep Lake#
This notebook showcases basic functionality related to Deep Lake. While Deep Lake can store embeddings, it is capable of storing any type of data. It is a fully fledged serverless data lake with version control, query engine and streaming dataloader to deep learning frameworks.
For more information, please see the Deep Lake documentation or api reference
!python3 -m pip install openai deeplake tiktoken
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.text_splitter import CharacterTextSplitter
from langchain.vectorstores import DeepLake
import os
import getpass
os.environ['OPENAI_API_KEY'] = getpass.getpass('OpenAI API Key:')
embeddings = OpenAIEmbeddings()
from langchain.document_loaders import TextLoader
loader = TextLoader('../../../state_of_the_union.txt')
documents = loader.load()
text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
docs = text_splitter.split_documents(documents)
embeddings = OpenAIEmbeddings()
db = DeepLake.from_documents(docs, embeddings)
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
mem://langchain loaded successfully.
Evaluating ingest: 100%|ââââââââââ| 1/1 [00:04<00:00
Dataset(path='mem://langchain', tensors=['embedding', 'ids', 'metadata', 'text'])
tensor htype shape dtype compression
------- ------- ------- ------- -------
embedding generic (4, 1536) float32 None
ids text (4, 1) str None
metadata json (4, 1) str None
text text (4, 1) str None
print(docs[0].page_content)
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâre at it, pass the Disclose Act so Americans can know who is funding our elections.
Tonight, Iâd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.
One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.
And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâs top legal minds, who will continue Justice Breyerâs legacy of excellence.
Retrieval Question/Answering#
from langchain.chains import RetrievalQA
from langchain.llms import OpenAIChat
qa = RetrievalQA.from_chain_type(llm=OpenAIChat(model='gpt-3.5-turbo'), chain_type='stuff', retriever=db.as_retriever())
/media/sdb/davit/.local/lib/python3.10/site-packages/langchain/llms/openai.py:624: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`
warnings.warn(
query = 'What did the president say about Ketanji Brown Jackson'
qa.run(query)
'The president nominated Circuit Court of Appeals Judge Ketanji Brown Jackson for the United States Supreme Court and praised her qualifications and broad support from both Democrats and Republicans.'
Attribute based filtering in metadata#
import random
for d in docs:
d.metadata['year'] = random.randint(2012, 2014)
db = DeepLake.from_documents(docs, embeddings)
mem://langchain loaded successfully.
Evaluating ingest: 100%|ââââââââââ| 1/1 [00:04<00:00
Dataset(path='mem://langchain', tensors=['embedding', 'ids', 'metadata', 'text'])
tensor htype shape dtype compression
------- ------- ------- ------- -------
embedding generic (42, 1536) float32 None
ids text (42, 1) str None
metadata json (42, 1) str None
text text (42, 1) str None
db.similarity_search('What did the president say about Ketanji Brown Jackson', filter={'year': 2013})
100%|ââââââââââ| 42/42 [00:00<00:00, 3456.17it/s]
[Document(page_content='A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since sheâs been nominated, sheâs received a broad range of supportâfrom the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \n\nAnd if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \n\nWe can do both. At our border, weâve installed new technology like cutting-edge scanners to better detect drug smuggling. \n\nWeâve set up joint patrols with Mexico and Guatemala to catch more human traffickers. \n\nWeâre putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster. \n\nWeâre securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
Document(page_content='And for our LGBTQ+ Americans, letâs finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong. \n\nAs I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. \n\nWhile it often appears that we never agree, that isnât true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice. \n\nAnd soon, weâll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things. \n\nSo tonight Iâm offering a Unity Agenda for the Nation. Four big things we can do together. \n\nFirst, beat the opioid epidemic.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
Document(page_content='Vice President Harris and I ran for office with a new economic vision for America. \n\nInvest in America. Educate Americans. Grow the workforce. Build the economy from the bottom up \nand the middle out, not from the top down. \n\nBecause we know that when the middle class grows, the poor have a ladder up and the wealthy do very well. \n\nAmerica used to have the best roads, bridges, and airports on Earth. \n\nNow our infrastructure is ranked 13th in the world. \n\nWe wonât be able to compete for the jobs of the 21st Century if we donât fix that. \n\nThatâs why it was so important to pass the Bipartisan Infrastructure Lawâthe most sweeping investment to rebuild America in history. \n\nThis was a bipartisan effort, and I want to thank the members of both parties who worked to make it happen. \n\nWeâre done talking about infrastructure weeks. \n\nWeâre going to have an infrastructure decade.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
Document(page_content='It is going to transform America and put us on a path to win the economic competition of the 21st Century that we face with the rest of the worldâparticularly with China. \n\nAs Iâve told Xi Jinping, it is never a good bet to bet against the American people. \n\nWeâll create good jobs for millions of Americans, modernizing roads, airports, ports, and waterways all across America. \n\nAnd weâll do it all to withstand the devastating effects of the climate crisis and promote environmental justice. \n\nWeâll build a national network of 500,000 electric vehicle charging stations, begin to replace poisonous lead pipesâso every childâand every Americanâhas clean water to drink at home and at school, provide affordable high-speed internet for every Americanâurban, suburban, rural, and tribal communities. \n\n4,000 projects have already been announced. \n\nAnd tonight, Iâm announcing that this year we will start fixing over 65,000 miles of highway and 1,500 bridges in disrepair.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013})]
Choosing distance function#
Distance function L2 for Euclidean, L1 for Nuclear, Max l-infinity distnace, cos for cosine similarity, dot for dot product
db.similarity_search('What did the president say about Ketanji Brown Jackson?', distance_metric='cos')
[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâre at it, pass the Disclose Act so Americans can know who is funding our elections. \n\nTonight, Iâd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâs top legal minds, who will continue Justice Breyerâs legacy of excellence.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012}),
Document(page_content='A former top litigator in private practice. A former federal public defender. And from a family of public school educators and police officers. A consensus builder. Since sheâs been nominated, sheâs received a broad range of supportâfrom the Fraternal Order of Police to former judges appointed by Democrats and Republicans. \n\nAnd if we are to advance liberty and justice, we need to secure the Border and fix the immigration system. \n\nWe can do both. At our border, weâve installed new technology like cutting-edge scanners to better detect drug smuggling. \n\nWeâve set up joint patrols with Mexico and Guatemala to catch more human traffickers. \n\nWeâre putting in place dedicated immigration judges so families fleeing persecution and violence can have their cases heard faster. \n\nWeâre securing commitments and supporting partners in South and Central America to host more refugees and secure their own borders.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
Document(page_content='And for our LGBTQ+ Americans, letâs finally get the bipartisan Equality Act to my desk. The onslaught of state laws targeting transgender Americans and their families is wrong. \n\nAs I said last year, especially to our younger transgender Americans, I will always have your back as your President, so you can be yourself and reach your God-given potential. \n\nWhile it often appears that we never agree, that isnât true. I signed 80 bipartisan bills into law last year. From preventing government shutdowns to protecting Asian-Americans from still-too-common hate crimes to reforming military justice. \n\nAnd soon, weâll strengthen the Violence Against Women Act that I first wrote three decades ago. It is important for us to show the nation that we can come together and do big things. \n\nSo tonight Iâm offering a Unity Agenda for the Nation. Four big things we can do together. \n\nFirst, beat the opioid epidemic.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2013}),
Document(page_content='Tonight, Iâm announcing a crackdown on these companies overcharging American businesses and consumers. \n\nAnd as Wall Street firms take over more nursing homes, quality in those homes has gone down and costs have gone up. \n\nThat ends on my watch. \n\nMedicare is going to set higher standards for nursing homes and make sure your loved ones get the care they deserve and expect. \n\nWeâll also cut costs and keep the economy going strong by giving workers a fair shot, provide more training and apprenticeships, hire them based on their skills not degrees. \n\nLetâs pass the Paycheck Fairness Act and paid leave. \n\nRaise the minimum wage to $15 an hour and extend the Child Tax Credit, so no one has to raise a family in poverty. \n\nLetâs increase Pell Grants and increase our historic support of HBCUs, and invest in what Jillâour First Lady who teaches full-timeâcalls Americaâs best-kept secret: community colleges.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2014})]
Maximal Marginal relevance#
Using maximal marginal relevance
db.max_marginal_relevance_search('What did the president say about Ketanji Brown Jackson?')
[Document(page_content='Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâre at it, pass the Disclose Act so Americans can know who is funding our elections. \n\nTonight, Iâd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \n\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \n\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâs top legal minds, who will continue Justice Breyerâs legacy of excellence.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012}),
Document(page_content='One was stationed at bases and breathing in toxic smoke from âburn pitsâ that incinerated wastes of warâmedical and hazard material, jet fuel, and more. \n\nWhen they came home, many of the worldâs fittest and best trained warriors were never the same. \n\nHeadaches. Numbness. Dizziness. \n\nA cancer that would put them in a flag-draped coffin. \n\nI know. \n\nOne of those soldiers was my son Major Beau Biden. \n\nWe donât know for sure if a burn pit was the cause of his brain cancer, or the diseases of so many of our troops. \n\nBut Iâm committed to finding out everything we can. \n\nCommitted to military families like Danielle Robinson from Ohio. \n\nThe widow of Sergeant First Class Heath Robinson. \n\nHe was born a soldier. Army National Guard. Combat medic in Kosovo and Iraq. \n\nStationed near Baghdad, just yards from burn pits the size of football fields. \n\nHeathâs widow Danielle is here with us tonight. They loved going to Ohio State football games. He loved building Legos with their daughter.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2014}),
Document(page_content='As Ohio Senator Sherrod Brown says, âItâs time to bury the label âRust Belt.â \n\nItâs time. \n\nBut with all the bright spots in our economy, record job growth and higher wages, too many families are struggling to keep up with the bills. \n\nInflation is robbing them of the gains they might otherwise feel. \n\nI get it. Thatâs why my top priority is getting prices under control. \n\nLook, our economy roared back faster than most predicted, but the pandemic meant that businesses had a hard time hiring enough workers to keep up production in their factories. \n\nThe pandemic also disrupted global supply chains. \n\nWhen factories close, it takes longer to make goods and get them from the warehouse to the store, and prices go up. \n\nLook at cars. \n\nLast year, there werenât enough semiconductors to make all the cars that people wanted to buy. \n\nAnd guess what, prices of automobiles went up. \n\nSoâwe have a choice. \n\nOne way to fight inflation is to drive down wages and make Americans poorer.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012}),
Document(page_content='We canât change how divided weâve been. But we can change how we move forwardâon COVID-19 and other issues we must face together. \n\nI recently visited the New York City Police Department days after the funerals of Officer Wilbert Mora and his partner, Officer Jason Rivera. \n\nThey were responding to a 9-1-1 call when a man shot and killed them with a stolen gun. \n\nOfficer Mora was 27 years old. \n\nOfficer Rivera was 22. \n\nBoth Dominican Americans whoâd grown up on the same streets they later chose to patrol as police officers. \n\nI spoke with their families and told them that we are forever in debt for their sacrifice, and we will carry on their mission to restore the trust and safety every community deserves. \n\nIâve worked on these issues a long time. \n\nI know what works: Investing in crime preventionand community police officers whoâll walk the beat, whoâll know the neighborhood, and who can restore trust and safety.', metadata={'source': '../../../state_of_the_union.txt', 'year': 2012})]
Deep Lake datasets on cloud (Activeloop, AWS, GCS, etc.) or local#
By default deep lake datasets are stored in memory, in case you want to persist locally or to any object storage you can simply provide path to the dataset. You can retrieve token from app.activeloop.ai
os.environ['ACTIVELOOP_TOKEN'] = getpass.getpass('Activeloop Token:')
# Embed and store the texts
dataset_path = f"hub://{USERNAME}/{DATASET_NAME}" # could be also ./local/path (much faster locally), s3://bucket/path/to/dataset, gcs://path/to/dataset, etc.
embedding = OpenAIEmbeddings()
vectordb = DeepLake.from_documents(documents=docs, embedding=embedding, dataset_path=dataset_path)
Your Deep Lake dataset has been successfully created!
The dataset is private so make sure you are logged in!
\
This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/davitbun/linkedin
hub://davitbun/linkedin loaded successfully.
Evaluating ingest: 100%|ââââââââââ| 1/1 [00:23<00:00
/
Dataset(path='hub://davitbun/linkedin', tensors=['embedding', 'ids', 'metadata', 'text'])
tensor htype shape dtype compression
------- ------- ------- ------- -------
embedding generic (42, 1536) float32 None
ids text (42, 1) str None
metadata json (42, 1) str None
text text (42, 1) str None
query = "What did the president say about Ketanji Brown Jackson"
docs = db.similarity_search(query)
print(docs[0].page_content)
Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while youâre at it, pass the Disclose Act so Americans can know who is funding our elections.
Tonight, Iâd like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyerâan Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service.
One of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court.
And I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nationâs top legal minds, who will continue Justice Breyerâs legacy of excellence.
vectordb.ds.summary()
Dataset(path='hub://davitbun/linkedin', tensors=['embedding', 'ids', 'metadata', 'text'])
tensor htype shape dtype compression
------- ------- ------- ------- -------
embedding generic (42, 1536) float32 None
ids text (42, 1) str None
metadata json (42, 1) str None
text text (42, 1) str None
embeddings = vectordb.ds.embedding.numpy()