Відео

Hi, Thank you so much for sharing the project video.. I am getting this error when I run the code in jupyter notebook in windows .....Py4JJavaError: An error occurred while calling o209.save. : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 58.0 failed 1 times, most recent failure: Lost task 0.0 in stage 58.0 (TID 46) (host.docker.internal executor driver): java.io.IOException: Mkdirs failed to create file:/C:/pythonwork/sparkwork/data/Apple Analysis/outputirpodsAfterIphone/_temporary/0/_temporary/attempt_202408032042443838464405057664188_0058_m_000000_46... can you please suggest?

For that python question: input = 'thebigdatashow' start = -1 * k end = len(input) - k for i in range(start,end): print(input[i],end="")

-- Create table statement CREATE TABLE orders ( order_id INT, product_id INT, quantity INT ); -- Insert data into the table INSERT INTO orders (order_id, product_id, quantity) VALUES (1, 1, 12), (1, 2, 10), (1, 3, 5), (1, 3, 10), (2, 1, 4), (2, 1, 4), (2, 4, 4), (2, 5, 6), (3, 3, 5), (3, 4, 18), (4, 5, 2), (4, 6, 8), (5, 7, 9), (5, 8, 9), (3, 9, 20), (2, 9, 4); For anybody practicing :)

Everyone talking about candidate.... She is excellent no doubt.. But. Interviewer helped her lead her to explain in dept

I really ljked this mock interview because usually the mock interviews are genrally go into this depth level. Appreciate the efforts put into this. Can we have more aws data engineering interview similarto this pledge?

This is what actual real world problem should be, love the video... This should reach more audience. PS: The strike rate of batsman is still incorrect, we have to include wide balls played by the batsman :)

Background music was unnecessary here.

Thanks

Hi, one question. why do we need to convert the dataframe to a dictionary when we can pass the dataframe directly to the transform function ??

Very informative ! 🤝

Does anybody have any idea what this error means: DBFS file browser StorageContext com.databricks.backend.storage.StorageContextType$DbfsRoot$@5c512926 for workspace 3667672304132597 is not set in the CustomerStorageInfo. I am not sure what this means. For context, I had uploaded some files yesterday on DBFS for practice purposes. Those files are still available if I try to find them through notebooks, however, the DBFS tab can't show them and throws this error. Any help will be much appreciated! EDIT: It was a Databricks global issue, has been fixed now.

Hey Manoj, great session on Data extraction via APIs. Few points I'd like to share from my experience working on this: 1) While dealing with huge volumes of data from source it's crucial to involve Pagination to iteratively collect all data. 2) Admins who manage these end_point URLs usually discourage having multiple API calls within a certain timeframe which would cause a dead-lock on your batch id. Like you suggested either have bulk data pulled all at once or optimize our framework.

Just completed this amazing project 😍 Can i add this in my portfolio?

I see this when trying to upload files using DBFS file browser Missing credentials to access AWS bucket

Thanks for the content, really beneficial.

Good efforts however please minimize the usage of words like "perfect" and "well & good" after every sentence

Great explanation thanks

subtitles are not fully visible

super manoj..txs for u time as you said its really tough usecase if possible do it with easy data set next time bcz every one can understand easliy keep going....👌

My Doubt is --> In spark we have cache and persist, used to save the RDD. As per my understanding cache and persist/MEMORY_AND_DISK both perform same action for DataFrames. If this is the case why should I prefer using cache at all, I can always use persist [with different parameters] and ignore cache. Could you please let me know, when to use cache, or if my understanding is wrong.

with tab1 as( SELECT order_id, max(quantity) as max_value, avg(quantity) as avg_quantity FROM orders GROUP BY order_id) SELECT order_id FROM tab1 WHERE max_value > ALL(SELECT avg_quantity FROM tab1 )

Thank you for your time and efforts. Its not a easy job 👏👏👏

important content is Questions asked 😅😅

Appreciate your great effort and share your knowledge brother!👍

please add the dataset

can you add the dataset link

Can you please help us with the dataset?

awesome sir

which pentab/device is used for video , can you please share ?

with cte as ( select order_id,AVG(quantity) as avg_quantity from t1 group by order_id ) select order_id from t1 where quantity > cte. avg_quantity and order_id=cte.order_id;

how come the project discussion is happening in the first round. Usually, they asked about Python, and SQL questions in the first round to check the basic foundation. Correct me if I am wrong here

Don't you think his intro was too long and the interviewer has to cut him in between to ask questions quickly?

Why would you setup a session in spark when you already get it ?

Replied in my previous question but it seems not visible so making again. Getting in first pipeline. AnalysisException: Failed to merge fields 'customer_id' and 'customer_id' Any suggestion would be appreciated. Thank you Please find the ap code I am trying to follow along here: github.com/maaz-ahmed-ansari/apple-product-analysis/tree/main

Getting in first pipeline. AnalysisException: Failed to merge fields 'customer_id' and 'customer_id' Any suggestion would be appreciated. Thank you

SELECT name, department_name, MAX(DATEDIFF(promotion_date, hire_date)) AS longest_time FROM employee JOIN department ON employee.dept_id = department.dept_id JOIN promotion ON employee.employee_id = promotion.employee_id GROUP BY department_name ORDER BY longest_time DESC;

This also increases the data size of the second dataset as we explode it, so is it still optimized as the data scan will be increased a lot and lot of shuffle will be involved?

A practical demonstration will be relaxed tomorrow. Kindly watch this video to understand the theory in depth.

Thank you , waiting

nums = [0,0,2,3,3,3,3,5,5] k = 2 new_nums = [] for i in nums: if nums.count(i) == k: if i not in new_nums: new_nums.append(i) print(new_nums)