How to remove duplicate rows in mysql without using temporary table

Spread the words...Share on Google+4Share on Facebook2Tweet about this on Twitter3Share on LinkedIn4Buffer this pageShare on StumbleUpon0Share on Reddit0

We as a developer, often come across situations where we have to work on database related stuffs. Mostly it is done when client send you its data in form of excel sheets and you push that data to database tables after some excel manipulations. I have also done it many times.

A very common problem faced in this approach is that it might result in duplicate rows at times, because data sent is mostly from departments like HR and finance where people are not well aware of data normalization techniques [:-)].

I will use Employee table where column names are id, name, department and email.

Below are the SQL scripts for generating the test data.

Create schema TestDB;

CREATE TABLE EMPLOYEE
(
    ID INT,
    NAME Varchar(100),
    DEPARTMENT INT,
    EMAIL Varchar(100)
);

INSERT INTO EMPLOYEE VALUES (1,'Anish',101,'anish@howtodoinjava.com');
INSERT INTO EMPLOYEE VALUES (2,'Lokesh',102,'lokesh@howtodoinjava.com');
INSERT INTO EMPLOYEE VALUES (3,'Rakesh',103,'rakesh@howtodoinjava.com');
INSERT INTO EMPLOYEE VALUES (4,'Yogesh',104,'yogesh@howtodoinjava.com');

--These are the duplicate rows

INSERT INTO EMPLOYEE VALUES (5,'Anish',101,'anish@howtodoinjava.com');
INSERT INTO EMPLOYEE VALUES (6,'Lokesh',102,'lokesh@howtodoinjava.com');

Solution:

DELETE e1 FROM EMPLOYEE e1, EMPLOYEE e2 WHERE e1.name = e2.name AND e1.id > e2.id;

Above sql query will delete rows where name field is duplicate and only those unique rows will be retained where name is unique and ID field is lowest. For example rows with ID 5 and 6 will be deleted and rows with 1 and 2 will be retained.

delete-duplicate-rows-in-mysql

If you want to retain rows with latest generated ID values, then reverse the condition in where clause to e1.id < e2.id like this:

DELETE e1 FROM EMPLOYEE e1, EMPLOYEE e2 WHERE e1.name = e2.name AND e1.id > e2.id;

If you want to compare multiple fields and add appropriate where clause.

Note: Please execute above (or modified) query first on test data always to make sure it is producing the expected output.

Happy Learning !!

Spread the words...Share on Google+4Share on Facebook2Tweet about this on Twitter3Share on LinkedIn4Buffer this pageShare on StumbleUpon0Share on Reddit0

 

 

Give me your email address and whenever I write any tutorial or discuss any java concept, you will receive a little email in your inbox.

Join 2,871 other subscribers

2 thoughts on “How to remove duplicate rows in mysql without using temporary table”

Want to ask any question? Or suggest anything?