Skip to content

Commit 7fb2563

Browse files
khuyentran1401khuyentran1401
committed
add content
1 parent 2cbba1f commit 7fb2563

File tree

13 files changed

+805
-165
lines changed

13 files changed

+805
-165
lines changed

‎Chapter1/dictionary.ipynb‎

Lines changed: 80 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -175,83 +175,112 @@
175175
"id": "5caf1b3f",
176176
"metadata": {},
177177
"source": [
178-
"### dict.get: Get the Default Value of a Dictionary if a Key Doesn't Exist"
178+
"### Stop Writing Nested if-else: Use Python's .get() Instead"
179179
]
180180
},
181181
{
182182
"cell_type": "markdown",
183-
"id": "c2dd1725",
183+
"id": "955af7b3-81c9-44d7-8b96-3113d7b3b199",
184184
"metadata": {},
185185
"source": [
186-
"If you want to get the default value when a key doesn't exist in a dictionary, use `dict.get`. In the code below, since there is no key `meeting3`, the default value `online` is returned. "
186+
"When working with dictionaries in Python, it's common to encounter situations where you need to access values that may or may not exist. The traditional approach of using multiple nested if-else statements can result in verbose, repetitive code that's harder to maintain and more prone to errors.\n",
187+
"\n",
188+
"Let's consider an example where we have a dictionary `user_data` with keys \"name\", \"age\", and possibly \"email\". We want to assign default values to these keys if they don't exist."
187189
]
188190
},
189191
{
190192
"cell_type": "code",
191-
"execution_count": 5,
192-
"id": "e066cd73",
193-
"metadata": {
194-
"ExecuteTime": {
195-
"end_time": "2021-09-30T12:49:56.062960Z",
196-
"start_time": "2021-09-30T12:49:56.056983Z"
193+
"execution_count": 2,
194+
"id": "88422abc-4db3-42fb-a655-938c2b1db0c4",
195+
"metadata": {},
196+
"outputs": [
197+
{
198+
"name": "stdout",
199+
"output_type": "stream",
200+
"text": [
201+
"name='Alice'\n",
202+
"age=30\n",
203+
"email='no-email@example.com'\n"
204+
]
197205
}
198-
},
199-
"outputs": [],
206+
],
200207
"source": [
201-
"locations = {'meeting1': 'room1', 'meeting2': 'room2'}"
208+
"# Checking dictionary values with multiple if-else\n",
209+
"user_data = {\"name\": \"Alice\", \"age\": 30}\n",
210+
"\n",
211+
"# Repetitive code with multiple default values\n",
212+
"if \"name\" in user_data:\n",
213+
" name = user_data[\"name\"]\n",
214+
"else:\n",
215+
" name = \"Unknown\"\n",
216+
" \n",
217+
"if \"age\" in user_data:\n",
218+
" age = user_data[\"age\"]\n",
219+
"else:\n",
220+
" age = 0\n",
221+
" \n",
222+
"if \"email\" in user_data:\n",
223+
" email = user_data[\"email\"]\n",
224+
"else:\n",
225+
" email = \"no-email@example.com\"\n",
226+
"\n",
227+
"\n",
228+
"print(f\"{name=}\")\n",
229+
"print(f\"{age=}\")\n",
230+
"print(f\"{email=}\")"
202231
]
203232
},
204233
{
205-
"cell_type": "code",
206-
"execution_count": 6,
207-
"id": "d07fcf3e",
208-
"metadata": {
209-
"ExecuteTime": {
210-
"end_time": "2021-09-30T12:49:59.099210Z",
211-
"start_time": "2021-09-30T12:49:59.090362Z"
212-
},
213-
"scrolled": true
214-
},
215-
"outputs": [
216-
{
217-
"data": {
218-
"text/plain": [
219-
"'room1'"
220-
]
221-
},
222-
"execution_count": 6,
223-
"metadata": {},
224-
"output_type": "execute_result"
225-
}
226-
],
234+
"cell_type": "markdown",
235+
"id": "21613f3c-afca-4d32-a158-8efd70503675",
236+
"metadata": {},
237+
"source": [
238+
"As you can see, this approach is tedious and prone to errors. "
239+
]
240+
},
241+
{
242+
"cell_type": "markdown",
243+
"id": "ea9eef2a-cd12-4439-82b7-cbfe84ccc42d",
244+
"metadata": {},
227245
"source": [
228-
"locations.get('meeting1', 'online')"
246+
"With the `.get()` method, we can access dictionary values with default values in a single line of code. This approach is not only more concise but also more readable and maintainable."
229247
]
230248
},
231249
{
232250
"cell_type": "code",
233251
"execution_count": 3,
234-
"id": "6de353f4",
235-
"metadata": {
236-
"ExecuteTime": {
237-
"end_time": "2021-09-30T12:49:46.738598Z",
238-
"start_time": "2021-09-30T12:49:46.729582Z"
239-
}
240-
},
252+
"id": "d97082d7-c441-4b7e-a011-a4f152ac92f0",
253+
"metadata": {},
241254
"outputs": [
242255
{
243-
"data": {
244-
"text/plain": [
245-
"'online'"
246-
]
247-
},
248-
"execution_count": 3,
249-
"metadata": {},
250-
"output_type": "execute_result"
256+
"name": "stdout",
257+
"output_type": "stream",
258+
"text": [
259+
"name='Alice'\n",
260+
"age=30\n",
261+
"email='no-email@example.com'\n"
262+
]
251263
}
252264
],
253265
"source": [
254-
"locations.get('meeting3', 'online')"
266+
"# Using .get() method for cleaner code\n",
267+
"user_data = {\"name\": \"Alice\", \"age\": 30}\n",
268+
"\n",
269+
"name = user_data.get(\"name\", \"Unknown\")\n",
270+
"age = user_data.get(\"age\", 0)\n",
271+
"email = user_data.get(\"email\", \"no-email@example.com\")\n",
272+
"\n",
273+
"print(f\"{name=}\")\n",
274+
"print(f\"{age=}\")\n",
275+
"print(f\"{email=}\")"
276+
]
277+
},
278+
{
279+
"cell_type": "markdown",
280+
"id": "c2dd1725",
281+
"metadata": {},
282+
"source": [
283+
"If you want to get the default value when a key doesn't exist in a dictionary, use `dict.get`. In the code below, since there is no key `meeting3`, the default value `online` is returned. "
255284
]
256285
},
257286
{

‎Chapter5/spark.ipynb‎

Lines changed: 193 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2000,6 +2000,199 @@
20002000
"SquareNumbers(lit(1), lit(3)).show()"
20012001
]
20022002
},
2003+
{
2004+
"cell_type": "markdown",
2005+
"id": "216e9fc0-12a9-4f45-85b1-8e791755b1d3",
2006+
"metadata": {},
2007+
"source": [
2008+
"### Best Practices for PySpark DataFrame Comparison Testing"
2009+
]
2010+
},
2011+
{
2012+
"cell_type": "code",
2013+
"execution_count": null,
2014+
"id": "9badb0ee-16ec-4291-9477-8a38ebd7e876",
2015+
"metadata": {
2016+
"editable": true,
2017+
"slideshow": {
2018+
"slide_type": ""
2019+
},
2020+
"tags": [
2021+
"hide-cell"
2022+
]
2023+
},
2024+
"outputs": [],
2025+
"source": [
2026+
"!pip install \"pyspark[sql]\""
2027+
]
2028+
},
2029+
{
2030+
"cell_type": "code",
2031+
"execution_count": 19,
2032+
"id": "d2adcd65-5197-404f-88d6-c368a863cf75",
2033+
"metadata": {
2034+
"editable": true,
2035+
"slideshow": {
2036+
"slide_type": ""
2037+
},
2038+
"tags": [
2039+
"hide-cell"
2040+
]
2041+
},
2042+
"outputs": [],
2043+
"source": [
2044+
"from pyspark.sql import SparkSession\n",
2045+
"\n",
2046+
"# Create SparkSession\n",
2047+
"spark = SparkSession.builder.getOrCreate()"
2048+
]
2049+
},
2050+
{
2051+
"cell_type": "markdown",
2052+
"id": "1002536a",
2053+
"metadata": {},
2054+
"source": [
2055+
"Manually comparing PySpark DataFrame outputs using `collect()` and equality comparison leads to brittle tests due to ordering issues and unclear error messages when data doesn't match expectations.\n",
2056+
"\n",
2057+
"For example, the following test will fail due to ordering issues, resulting in an unclear error message.\n"
2058+
]
2059+
},
2060+
{
2061+
"cell_type": "code",
2062+
"execution_count": 31,
2063+
"id": "e4299f30",
2064+
"metadata": {},
2065+
"outputs": [
2066+
{
2067+
"name": "stderr",
2068+
"output_type": "stream",
2069+
"text": [
2070+
" \r"
2071+
]
2072+
},
2073+
{
2074+
"name": "stdout",
2075+
"output_type": "stream",
2076+
"text": [
2077+
"assert [Row(id=1, name='Alice', value=100), Row(id=2, name='Bob', value=200)] == [Row(id=2, name='Bob', value=200), Row(id=1, name='Alice', value=100)]\n",
2078+
" + where [Row(id=1, name='Alice', value=100), Row(id=2, name='Bob', value=200)] = <bound method DataFrame.collect of DataFrame[id: bigint, name: string, value: bigint]>()\n",
2079+
" + where <bound method DataFrame.collect of DataFrame[id: bigint, name: string, value: bigint]> = DataFrame[id: bigint, name: string, value: bigint].collect\n",
2080+
" + and [Row(id=2, name='Bob', value=200), Row(id=1, name='Alice', value=100)] = <bound method DataFrame.collect of DataFrame[id: bigint, name: string, value: bigint]>()\n",
2081+
" + where <bound method DataFrame.collect of DataFrame[id: bigint, name: string, value: bigint]> = DataFrame[id: bigint, name: string, value: bigint].collect\n"
2082+
]
2083+
}
2084+
],
2085+
"source": [
2086+
"# Manual DataFrame comparison\n",
2087+
"result_df = spark.createDataFrame(\n",
2088+
" [(1, \"Alice\", 100), (2, \"Bob\", 200)], [\"id\", \"name\", \"value\"]\n",
2089+
")\n",
2090+
"\n",
2091+
"expected_df = spark.createDataFrame(\n",
2092+
" [(2, \"Bob\", 200), (1, \"Alice\", 100)], [\"id\", \"name\", \"value\"]\n",
2093+
")\n",
2094+
"\n",
2095+
"try:\n",
2096+
" assert result_df.collect() == expected_df.collect()\n",
2097+
"except AssertionError as e:\n",
2098+
" print(e)"
2099+
]
2100+
},
2101+
{
2102+
"cell_type": "markdown",
2103+
"id": "7c4f8fd8-c2c2-4804-8e42-6fd3eb6aec27",
2104+
"metadata": {},
2105+
"source": [
2106+
"`assertDataFrameEqual` provides a robust way to compare DataFrames, allowing for order-independent comparison.\n"
2107+
]
2108+
},
2109+
{
2110+
"cell_type": "code",
2111+
"execution_count": null,
2112+
"id": "73b0d483-8b00-44ab-9279-4c7765ca1ff6",
2113+
"metadata": {},
2114+
"outputs": [],
2115+
"source": [
2116+
"# Testing with DataFrame equality\n",
2117+
"from pyspark.testing.utils import assertDataFrameEqual"
2118+
]
2119+
},
2120+
{
2121+
"cell_type": "code",
2122+
"execution_count": 7,
2123+
"id": "7c46ae8a",
2124+
"metadata": {},
2125+
"outputs": [],
2126+
"source": [
2127+
"assertDataFrameEqual(result_df, expected_df)"
2128+
]
2129+
},
2130+
{
2131+
"cell_type": "markdown",
2132+
"id": "085f150d-20ff-4b0a-a4ab-1ee452598e9e",
2133+
"metadata": {},
2134+
"source": [
2135+
"Using `collect()` for comparison cannot detect type mismatch, whereas `assertDataFrameEqual` can.\n",
2136+
"\n",
2137+
"For example, the following test will pass, even though there is a type mismatch.\n"
2138+
]
2139+
},
2140+
{
2141+
"cell_type": "code",
2142+
"execution_count": 27,
2143+
"id": "da7494c0-c05f-4a2f-a411-805c8f2f73ba",
2144+
"metadata": {},
2145+
"outputs": [],
2146+
"source": [
2147+
"# Manual DataFrame comparison\n",
2148+
"result_df = spark.createDataFrame(\n",
2149+
" [(1, \"Alice\", 100), (2, \"Bob\", 200)], [\"id\", \"name\", \"value\"]\n",
2150+
")\n",
2151+
"\n",
2152+
"expected_df = spark.createDataFrame(\n",
2153+
" [(1, \"Alice\", 100.0), (2, \"Bob\", 200.0)], [\"id\", \"name\", \"value\"]\n",
2154+
")\n",
2155+
"\n",
2156+
"assert result_df.collect() == expected_df.collect()"
2157+
]
2158+
},
2159+
{
2160+
"cell_type": "markdown",
2161+
"id": "82914b3b-69c0-4c68-9d72-d2ce31417397",
2162+
"metadata": {},
2163+
"source": [
2164+
"The error message produced by `assertDataFrameEqual` is clear and informative, highlighting the difference in schemas."
2165+
]
2166+
},
2167+
{
2168+
"cell_type": "code",
2169+
"execution_count": 30,
2170+
"id": "3faa1dbc-887a-4c36-ace8-c621411c3fb7",
2171+
"metadata": {},
2172+
"outputs": [
2173+
{
2174+
"name": "stdout",
2175+
"output_type": "stream",
2176+
"text": [
2177+
"[DIFFERENT_SCHEMA] Schemas do not match.\n",
2178+
"--- actual\n",
2179+
"+++ expected\n",
2180+
"- StructType([StructField('id', LongType(), True), StructField('name', StringType(), True), StructField('value', LongType(), True)])\n",
2181+
"? ^ ^^\n",
2182+
"\n",
2183+
"+ StructType([StructField('id', LongType(), True), StructField('name', StringType(), True), StructField('value', DoubleType(), True)])\n",
2184+
"? ^ ^^^^\n",
2185+
"\n"
2186+
]
2187+
}
2188+
],
2189+
"source": [
2190+
"try:\n",
2191+
" assertDataFrameEqual(result_df, expected_df)\n",
2192+
"except AssertionError as e:\n",
2193+
" print(e)"
2194+
]
2195+
},
20032196
{
20042197
"cell_type": "markdown",
20052198
"id": "9da7e800",

‎docs/Chapter1/Chapter1.html‎

Lines changed: 18 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -272,7 +272,24 @@
272272
<li class="toctree-l2"><a class="reference internal" href="../Chapter5/testing.html">6.13. Testing</a></li>
273273
<li class="toctree-l2"><a class="reference internal" href="../Chapter5/SQL.html">6.14. SQL Libraries</a></li>
274274
<li class="toctree-l2"><a class="reference internal" href="../Chapter5/spark.html">6.15. 3 Powerful Ways to Create PySpark DataFrames</a></li>
275-
<li class="toctree-l2"><a class="reference internal" href="../Chapter5/llm.html">6.16. Large Language Model (LLM)</a></li>
275+
276+
277+
278+
279+
280+
281+
282+
283+
284+
285+
286+
287+
288+
289+
290+
291+
292+
<li class="toctree-l2"><a class="reference internal" href="../Chapter5/llm.html">6.33. Large Language Model (LLM)</a></li>
276293
</ul>
277294
</li>
278295
<li class="toctree-l1 has-children"><a class="reference internal" href="../Chapter6/Chapter6.html">7. Cool Tools</a><input class="toctree-checkbox" id="toctree-checkbox-7" name="toctree-checkbox-7" type="checkbox"/><label class="toctree-toggle" for="toctree-checkbox-7"><i class="fa-solid fa-chevron-down"></i></label><ul>

0 commit comments

Comments
 (0)