Re: [HPDD-discuss] Large number of sync requests
by Dilger, Andreas
What does strace of the application show? The clients also keep track of sync operations:
lctl get_param {osc,mdc}.*.stats | grep sync
and also the time it takes these operations to complete. For my OSTs it is about 25ms (SATA HDD), and the MDT it is about 3ms (old SSD).
If the application is issuing a lot of sync requests, not much that can be done about it except fix the app.
Cheers, Andreas
--
Andreas Dilger
Lustre Principal Architect
Intel High Performance Data Division
On 2016/02/17, 09:16, "HPDD-discuss on behalf of Kumar, Amit" <hpdd-discuss-bounces(a)lists.01.org<mailto:hpdd-discuss-bounces@lists.01.org> on behalf of ahkumar(a)mail.smu.edu<mailto:ahkumar@mail.smu.edu>> wrote:
Dear All,
I am noticing large number of sync requests originating from the jobs. And I am trying to understand the numbers.
I handpicked a few below from the output of “lctl get_param mdt.scratch-MDT0000.job_stats” from MDS. Strangely when I run try to get job_stats from all OSS for all OST’s I don’t see these high sync requests, rather it is 0.
My question why would I see these many sync requests, even if it is cumulative it is high for a job that runs for couple of hours at the most.
Can any body please shed some light on what I am seeing here?
Thank you,
Amit
- job_id: 7819402
snapshot_time: 1455722134
open: { samples: 632, unit: reqs }
close: { samples: 484, unit: reqs }
mknod: { samples: 0, unit: reqs }
link: { samples: 0, unit: reqs }
unlink: { samples: 0, unit: reqs }
mkdir: { samples: 0, unit: reqs }
rmdir: { samples: 0, unit: reqs }
rename: { samples: 0, unit: reqs }
getattr: { samples: 58, unit: reqs }
setattr: { samples: 0, unit: reqs }
getxattr: { samples: 0, unit: reqs }
setxattr: { samples: 0, unit: reqs }
statfs: { samples: 0, unit: reqs }
sync: { samples: 1183934, unit: reqs }
samedir_rename: { samples: 0, unit: reqs }
crossdir_rename: { samples: 0, unit: reqs }
- job_id: 7819409
snapshot_time: 1455722644
open: { samples: 738, unit: reqs }
close: { samples: 572, unit: reqs }
mknod: { samples: 0, unit: reqs }
link: { samples: 0, unit: reqs }
unlink: { samples: 0, unit: reqs }
mkdir: { samples: 0, unit: reqs }
rmdir: { samples: 0, unit: reqs }
rename: { samples: 0, unit: reqs }
getattr: { samples: 66, unit: reqs }
setattr: { samples: 0, unit: reqs }
getxattr: { samples: 0, unit: reqs }
setxattr: { samples: 0, unit: reqs }
statfs: { samples: 0, unit: reqs }
sync: { samples: 1396655, unit: reqs }
samedir_rename: { samples: 0, unit: reqs }
crossdir_rename: { samples: 0, unit: reqs }
6 years, 7 months
Top 5 Reasons You CAN'T Miss LUG 2016
by OpenSFS Administration
<http://opensfs.org/events/lug-2016/>
LUSTRE USER GROUP 2016
Top 5 Reasons To Attend LUG 2016
On April 5-7th, OpenSFS is hosting the
<http://opensfs.org/events/lug-2016/> 14th annual LustreR User Group (LUG)
conference at the
<http://www3.hilton.com/en/hotels/oregon/hilton-portland-and-executive-tower
-PDXPHHH/index.html> Hilton Portland and Executive Tower in Portland,
Oregon! LUG 2016 will bring together industry leaders, end-users,
developers, and vendors to talk Lustre, contribute to the community, and
move the technology forward.
Here are the top 5 reasons why you should attend LUG 2016:
5 - Free Beer!
LUG wouldn't be LUG without free beer! One of the highlights of LUG is the
opportunity to catch up with old Lustre friends and make new Lustre friends.
It's even better with free beer from local Portland breweries. Join us on
Tuesday, April 6th for a networking reception with sweeping views of the
Portland skyline. And free beer.
4 - Network with Your Peers and Connect with Key Influencers
Meet and mingle with the people who "speak Lustre." LUG 2016 is your chance
to network, collaborate and share insight with developers, end-users, and
other members of the Lustre community over meals, during breaks, and at a
special evening reception.
3 - Stay Current on Feature Developments
Dynamic sessions from engineers working on features for upcoming community
releases of Lustre.
2 - Visit the City of Roses
The venue for LUG 2016 is the
<http://www3.hilton.com/en/hotels/oregon/hilton-portland-and-executive-tower
-PDXPHHH/index.html> Hilton Portland and Executive Tower, located in the
heart of Downtown Portland. Take in all that Portland has to offer,
including a thriving <http://www.foodcartsportland.com/maps/> food cart
scene, <http://www.travelportland.com/article/portland-breweries/> over 60
craft breweries in the city limits, and much
<http://www.tripadvisor.com/Attractions-g52024-Activities-Portland_Oregon.ht
ml> more!
1 - High Bandwidth, Low Latency Connections to Lustre Developers, Admins,
and Users
Prior attendees know that LUG is a firehouse of Lustre information. Get
caught up on what's going on with Lustre, talk with others about Lustre
challenges you are facing, understand how others are using Lustre, and
what's coming down the road.
Lustre Developer Day
Want more Lustre or a chance to talk specifically with Developers? Be sure
to arrive a day early for the Lustre Developer Day. Developers looking for a
focused opportunity to present their ideas, share current work, and learn
from others will want to attend our 2nd annual Lustre Developer Day on April
4th. <http://opensfs.org/events/lustre-developer-day/> Learn More!
If you have any questions, please feel free to contact
<mailto:admin@opensfs.org> admin(a)opensfs.org.
We look forward to seeing you in Portland at LUG 2016!
REGISTRATION
Register by March 11th to take advantage of the special early-bird
registration rate!
Early Bird Registration = $525
Regular Registration = $575
<http://events.r20.constantcontact.com/register/event?oeidk=a07ec4atvk2ca30d
f75&llr=c9ldpiwab>
\
Open Scalable File Systems, Inc. was founded in 2010 to advance Lustre
development, ensuring it remains vendor-neutral, open, and free. Since its
inception, OpenSFS has been responsible for advancing the Lustre file system
and delivering new releases on behalf of the open source community. Through
working groups, events, and ongoing funding initiatives, OpenSFS harnesses
the power of collaborative development to fuel innovation and growth of the
Lustre file system worldwide.
OpenSFS Administration
3855 SW 153rd Drive Beaverton, OR 97003 USA | Phone: +1 503-619-0561 | Fax:
+1 503-644-6708
Twitter: <https://twitter.com/opensfs> @OpenSFS | Email:
<mailto:admin@opensfs.org> admin(a)opensfs.org | Website:
<http://www.opensfs.org> www.opensfs.org
6 years, 7 months
Large number of sync requests
by Kumar, Amit
Dear All,
I am noticing large number of sync requests originating from the jobs. And I am trying to understand the numbers.
I handpicked a few below from the output of "lctl get_param mdt.scratch-MDT0000.job_stats" from MDS. Strangely when I run try to get job_stats from all OSS for all OST's I don't see these high sync requests, rather it is 0.
My question why would I see these many sync requests, even if it is cumulative it is high for a job that runs for couple of hours at the most.
Can any body please shed some light on what I am seeing here?
Thank you,
Amit
- job_id: 7819402
snapshot_time: 1455722134
open: { samples: 632, unit: reqs }
close: { samples: 484, unit: reqs }
mknod: { samples: 0, unit: reqs }
link: { samples: 0, unit: reqs }
unlink: { samples: 0, unit: reqs }
mkdir: { samples: 0, unit: reqs }
rmdir: { samples: 0, unit: reqs }
rename: { samples: 0, unit: reqs }
getattr: { samples: 58, unit: reqs }
setattr: { samples: 0, unit: reqs }
getxattr: { samples: 0, unit: reqs }
setxattr: { samples: 0, unit: reqs }
statfs: { samples: 0, unit: reqs }
sync: { samples: 1183934, unit: reqs }
samedir_rename: { samples: 0, unit: reqs }
crossdir_rename: { samples: 0, unit: reqs }
- job_id: 7819409
snapshot_time: 1455722644
open: { samples: 738, unit: reqs }
close: { samples: 572, unit: reqs }
mknod: { samples: 0, unit: reqs }
link: { samples: 0, unit: reqs }
unlink: { samples: 0, unit: reqs }
mkdir: { samples: 0, unit: reqs }
rmdir: { samples: 0, unit: reqs }
rename: { samples: 0, unit: reqs }
getattr: { samples: 66, unit: reqs }
setattr: { samples: 0, unit: reqs }
getxattr: { samples: 0, unit: reqs }
setxattr: { samples: 0, unit: reqs }
statfs: { samples: 0, unit: reqs }
sync: { samples: 1396655, unit: reqs }
samedir_rename: { samples: 0, unit: reqs }
crossdir_rename: { samples: 0, unit: reqs }
6 years, 7 months
Last Chance to Submit an Abstract for LUG 2016
by OpenSFS Administration
<http://opensfs.org/>
LUSTRE USER GROUP
LUG 2016
CONFERENCE
LUSTRE MATTERS.
APRIL 5-7, 2016
HILTON PORTLAND AND EXECUTIVE TOWER
PORTLAND, OR
CALL FOR PRESENTATIONS
CALLING ALL END-USERS, ADMINISTRATORS & DEVELOPERS OF LUSTRE.
Friday, February 12th is the final day to submit a presentation for the
<http://opensfs.org/events/lug-2016/> 14th annual LustreR User Group (LUG)
conference!
SUBMISSION INSTRUCTIONS
To submit an abstract please visit the
<https://easychair.org/conferences/?conf=lug2016> LUG 2016 Call for
Presentation Submission Website: Abstracts should be a minimum of 250 words
and should provide a clear description of the planned presentation and its
goals.
* You will need to create a user account on Easy Chair (our online
submission platform) if you don't already have one
* After providing your user details via Easy Chair, you will be
prompted to complete the online submission process
Accepted presentations will be subject to panel review three weeks in
advance of LUG 2016.
Additional submission guidelines are available on the
<http://opensfs.org/call-for-participation/> Call for Participation website.
REGISTRATION
Register by March 11th to take advantage of the special early-bird
registration rate!
Early Bird Registration = $525
Regular Registration = $575
<http://events.r20.constantcontact.com/register/event?oeidk=a07ec4atvk2ca30d
f75&llr=c9ldpiwab>
LUSTER DEVELOPER DAY
Developers looking for a focused opportunity to present their ideas, share
current work, and learn from others will want to attend our 2nd annual
Lustre Developer Day on April 4th. Learn More!
<http://opensfs.org/events/lustre-developer-day/>
Open Scalable File Systems, Inc. was founded in 2010 to advance Lustre
development, ensuring it remains vendor-neutral, open, and free. Since its
inception, OpenSFS has been responsible for advancing the Lustre file system
and delivering new releases on behalf of the open source community. Through
working groups, events, and ongoing funding initiatives, OpenSFS harnesses
the power of collaborative development to fuel innovation and growth of the
Lustre file system worldwide.
OpenSFS Administration
3855 SW 153rd Drive Beaverton, OR 97003 USA | Phone: +1 503-619-0561 | Fax:
+1 503-644-6708
Twitter: <https://twitter.com/opensfs> @OpenSFS | Email:
<mailto:admin@opensfs.org> admin(a)opensfs.org | Website:
<http://www.opensfs.org> www.opensfs.org
6 years, 7 months
LUG 2016: Friday Abstract Submission Deadline
by OpenSFS Administration
<http://opensfs.org/events/lug-2016/>
LUG 2016: Call For Participation
Abstracts Due Friday, Feb 12th
Friday, February 12th is the final day to submit a presentation for the
<http://opensfs.org/events/lug-2016/> 14th annual LustreR User Group (LUG)
conference!
The abstract submission deadline is February 12, 2016
No submission extensions will be granted
The OpenSFS LUG Planning Committee is looking for presentations related to
specific development challenges, new feature development, project updates,
edge-case implementations, and more. Does this sound like something you are
working on? Then submit an abstract today!
Submit an Abstract
The over-arching goal of LUG 2016 is to provide a forum for end user sites
to share knowledge and experience with their peers. All LUG presentations
will be 30 minutes, with a total of 25 speaking opportunities available.
To submit an abstract please visit the
<https://easychair.org/conferences/?conf=lug2016> LUG 2016 Call for
Presentation Submission Website: Your abstract should be a minimum of 250
words and should provide a clear description of the planned presentation and
its goals.
* You will need to create a user account on Easy Chair (our online
submission platform) if you don't already have one
* After providing your user details via Easy Chair, you will be
prompted to complete the online submission process
Please note, the LUG Program Committee will prioritize abstracts from
end-users and their organizations. Accepted presentations will be subject to
panel review three weeks in advance of LUG 2016.
Additional submission guidelines are available on the
<http://opensfs.org/call-for-participation/> Call for Participation website.
Event Overview
LUG 2016 is the primary event for technical presentations and discussions on
the Lustre parallel file system. This year's conference is being held in
Portland, OR, from April 5-7, 2016.
<http://events.r20.constantcontact.com/register/event?oeidk=a07ec4atvk2ca30d
f75&llr=c9ldpiwab> Register now to take advantage of the special early bird
rate before March 11th. The early bird registration price is $525.
We look forward to seeing you in Portland at LUG 2016!
Best regards,
OpenSFS LUG Planning Committee
<http://events.r20.constantcontact.com/register/event?oeidk=a07ec4atvk2ca30d
f75&llr=c9ldpiwab>
<http://opensfs.org/events/lug-2016/lug-2016-sponsorship/> Click here to
learn how to become a 2016 LUG Sponsor
Open Scalable File Systems, Inc. was founded in 2010 to advance Lustre
development, ensuring it remains vendor-neutral, open, and free. Since its
inception, OpenSFS has been responsible for advancing the Lustre file system
and delivering new releases on behalf of the open source community. Through
working groups, events, and ongoing funding initiatives, OpenSFS harnesses
the power of collaborative development to fuel innovation and growth of the
Lustre file system worldwide.
OpenSFS Administration
3855 SW 153rd Drive Beaverton, OR 97003 USA | Phone: +1 503-619-0561 | Fax:
+1 503-644-6708
Twitter: <https://twitter.com/opensfs> @OpenSFS | Email:
<mailto:admin@opensfs.org> admin(a)opensfs.org | Website:
<http://www.opensfs.org> www.opensfs.org
6 years, 7 months
Register Today: LUG 2016
by OpenSFS Administration
<http://opensfs.org/events/lug-2016/>
LustreR User Group Conference 2016
Registration Now Open
Would you like to hear from the world's leading developers, administrators,
solution providers, and users of Lustre? Register now for the
<http://opensfs.org/events/lug-2016/> 14th annual LustreR User Group (LUG)
conference!
April 5-7, 2016
<http://www3.hilton.com/en/hotels/oregon/hilton-portland-and-executive-tower
-PDXPHHH/index.html> Hilton Portland and Executive Tower
921 SW Sixth Avenue, Portland, OR 97204
<http://opensfs.org/events/lug-2016/> LUG 2016 is the industry's primary
venue for discussion and seminars on the Lustre parallel file system and
other open source file system technologies. Don't miss your chance to
actively participate in industry dialogue on best practices and emerging
technologies, explore upcoming developments of the Lustre file system, and
immerse in the strong Lustre community.
We are also happy to offer one complimentary event pass to each member
company of OpenSFS as a benefit of membership. The registration discount
code will be sent to the primary contact, but it may be transferred to
another representative at the same company if the primary contact will not
be attending.
Register now to take advantage of the special early bird rate before March
11th. The early bird registration price is $525.
<http://events.r20.constantcontact.com/register/event?oeidk=a07ec4atvk2ca30d
f75&llr=c9ldpiwab>
Be sure to check back regularly, as speakers and session highlights will
also be posted over the coming weeks.
Lustre Developer Day
Join us for the 2nd annual Lustre Developer Day on Monday, April 4.
Developers looking for a focused opportunity to present their ideas, share
current work, and learn from others will not want to miss this one-day
event! The Developer Day will provide newer developers with an opportunity
to listen, learn, and network while providing veteran developers with a
forum for socializing current work and issues.
More information about the Developer Day, including agenda and registration,
is available on the <http://opensfs.org/events/lustre-developer-day/>
Developer Day webpage.
<http://events.r20.constantcontact.com/register/event?oeidk=a07ec42zs3207cd2
5f2&llr=c9ldpiwab> >> Register for Lustre Developer Day 2016 <<
We look forward to seeing you in Portland. Please feel free to contact us if
you have any questions.
Best regards,
OpenSFS LUG Planning Committee
Call for Presentations
Interested in shaping the LUG 2016 agenda? The Planning Committee is
accepting presentation proposals until February 12!
Learn how to submit an abstract by visiting the
<http://opensfs.org/call-for-participation/> Call for Participation Webpage.
<http://opensfs.org/events/lug-2016/lug-2016-sponsorship/> Click here to
learn how to become a 2016 LUG Sponsor
Open Scalable File Systems, Inc. was founded in 2010 to advance Lustre
development, ensuring it remains vendor-neutral, open, and free. Since its
inception, OpenSFS has been responsible for advancing the Lustre file system
and delivering new releases on behalf of the open source community. Through
working groups, events, and ongoing funding initiatives, OpenSFS harnesses
the power of collaborative development to fuel innovation and growth of the
Lustre file system worldwide.
OpenSFS Administration
3855 SW 153rd Drive Beaverton, OR 97003 USA | Phone: +1 503-619-0561 | Fax:
+1 503-644-6708
Twitter: <https://twitter.com/opensfs> @OpenSFS | Email:
<mailto:admin@opensfs.org> admin(a)opensfs.org | Website:
<http://www.opensfs.org> www.opensfs.org
6 years, 8 months