You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri May 09, 2025 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30859210490,9sleep10-21swapper/119:08:161
30987210088,8sleep30-21swapper/319:09:553
3082029070,16sleep20-21swapper/219:07:452
3070428270,8sleep00-21swapper/019:06:140
3108099290,10cyclictest12470-21aten2_r7power_e22:15:151
3108699270,2cyclictest0-21swapper/321:20:003
3108399261,24cyclictest0-21swapper/220:30:212
3108099254,1cyclictest23554-21diskmemload22:25:201
31083992420,3cyclictest676-21ntp_states23:20:262
3108399240,23cyclictest0-21swapper/220:00:202
31086992320,2cyclictest31832-21apache_volume23:20:183
31083992320,2cyclictest26125-21ntp_states21:35:222
31083992320,2cyclictest23567-21munin-run21:10:002
31083992320,2cyclictest23567-21munin-run21:09:592
3108099235,8cyclictest0-21swapper/122:30:131
3108099230,0cyclictest0-21swapper/123:42:111
31076992320,2cyclictest31866-21apache_volume22:55:140
31076992320,2cyclictest18400-21apache_volume21:30:130
31076992320,2cyclictest14331-21users22:40:280
276322230,4chrt39-21ksoftirqd/322:01:033
31083992219,2cyclictest5959-21hddtemp_smartct21:20:202
31083992219,2cyclictest19067-21apache_accesses23:10:112
3108099225,11cyclictest0-21swapper/120:25:151
31076992219,2cyclictest418-21memory23:45:210
31076992219,2cyclictest11351-21irqstats00:20:190
3107699220,19cyclictest0-21swapper/019:55:130
3108099210,19cyclictest0-21swapper/100:34:461
31076992118,2cyclictest6084-21apache_volume22:35:140
31076992118,2cyclictest4269-21apache_accesses00:15:150
240502210,0sleep10-21swapper/121:34:371
31086992017,2cyclictest32468-21df22:30:153
3108699200,1cyclictest0-21swapper/323:30:163
3108699200,19cyclictest0-21swapper/321:24:253
3108399200,18cyclictest0-21swapper/221:14:172
31080992017,2cyclictest26628-21ntp_states23:15:201
31076992017,2cyclictest24020-21apache_accesses21:10:130
31086991917,1cyclictest336-21ntp_states21:40:223
31083991918,1cyclictest0-21swapper/200:15:172
31083991918,1cyclictest0-21swapper/200:08:342
31080991912,2cyclictest7102-21ntp_states22:10:211
3108099191,14cyclictest0-21swapper/122:35:151
31080991910,3cyclictest25416-21grep22:50:151
31080991910,2cyclictest20446-21tr23:10:211
31076991916,2cyclictest7759-21users22:10:270
31076991914,4cyclictest16942-21python19:45:260
3108699180,1cyclictest0-21swapper/321:39:083
31083991815,2cyclictest26568-21memory22:25:212
31083991815,2cyclictest14329-21users22:15:282
31083991814,3cyclictest23554-21diskmemload23:51:112
3108099189,3cyclictest12891-21sh19:40:001
31080991812,2cyclictest7029-21users21:20:281
3108099180,1cyclictest0-21swapper/122:57:321
3108099180,1cyclictest0-21swapper/122:47:571
3107699188,5cyclictest18213-21perf21:55:000
3107699186,6cyclictest5781-21munin-run19:25:000
31086991714,2cyclictest25252-21apache_processe23:40:143
31086991713,3cyclictest381rcuc/323:35:143
3108399171,1cyclictest186402chrt23:35:032
31080991714,2cyclictest5836-21apache_volume23:00:131
3108099170,1cyclictest0-21swapper/123:45:241
3108099170,12cyclictest10544-21wc19:35:011
3107699172,8cyclictest0-21swapper/022:00:420
31076991714,2cyclictest12549-21hddtemp_smartct21:25:240
31086991616,0cyclictest0-21swapper/322:50:253
31083991615,1cyclictest23554-21diskmemload22:50:362
3108399160,1cyclictest23554-21diskmemload23:15:122
3108099168,3cyclictest18341-21wc21:55:001
31080991613,2cyclictest31463-21df_inode21:15:171
3108099160,16cyclictest0-21swapper/122:08:491
31076991613,2cyclictest24412-21munin-run00:05:000
31086991513,1cyclictest7622-21users23:50:253
31086991512,2cyclictest13081-21diskstats22:40:163
31086991511,3cyclictest1142-21chrt00:11:183
3108699150,14cyclictest0-21swapper/322:07:543
3108399158,6cyclictest12498-21grep21:25:232
31083991513,1cyclictest32574-21diskstats22:30:162
3108399151,2cyclictest37850irq/131-snd_hda00:30:272
31083991512,2cyclictest10859-21df00:20:152
3108099152,8cyclictest18392-21turbostat23:35:011
31080991510,3cyclictest19137-21taskset21:00:161
3107699151,7cyclictest5613-21perf23:25:010
3108699149,3cyclictest0-21swapper/323:46:193
3108699148,5cyclictest30065-21apache221:52:493
3108699145,5cyclictest19032-21/usr/sbin/munin23:10:153
31086991414,0cyclictest0-21swapper/323:16:223
31083991413,1cyclictest0-21swapper/223:02:362
31083991410,3cyclictest10511-21idleruntime-cro19:35:012
3108399140,13cyclictest22208-21head19:55:262
3108099149,3cyclictest21557-21wc19:55:191
3108099146,7cyclictest10982-21munin-node00:12:431
31080991410,2cyclictest12534-21sh22:40:151
3108099140,12cyclictest7480-21cat19:25:271
3107699143,9cyclictest31449-21perf22:55:000
31076991411,2cyclictest12402-21apache_accesses23:30:140
31076991410,2cyclictest12521-21grep23:55:160
3108699139,3cyclictest18292-21grep20:55:273
3108699139,3cyclictest12902-21munin-plugin-st19:40:013
3108699139,2cyclictest12893-21cat20:45:213
3108699133,5cyclictest224-21systemd-journal00:09:593
31086991310,2cyclictest21923-21tr00:30:193
31086991310,2cyclictest2047-21head19:15:183
31086991310,2cyclictest12872-21df_inode23:55:183
31086991310,2cyclictest11666-21apt20:44:593
3108399139,3cyclictest5822-21grep21:45:152
3108399139,3cyclictest31851-21ssh22:55:122
3108399139,3cyclictest13796-21wc22:40:232
3108399139,2cyclictest27421-21grep00:35:212
3108399139,2cyclictest24408-21munin-plugin-st00:05:002
3108399139,2cyclictest18370-21grep20:55:292
3108399138,4cyclictest15308-21sh23:06:292
31083991310,2cyclictest5407-21cron22:10:002
31083991310,2cyclictest17458-21uname00:25:222
31083991310,1cyclictest0-21swapper/221:17:112
3108399130,13cyclictest0-21swapper/222:38:382
3108099139,3cyclictest3727-21grep19:20:111
3108099138,3cyclictest18780-21ls21:00:001
3108099135,3cyclictest6113-21ssh23:25:161
31080991310,2cyclictest26296-21wc22:00:221
31080991310,2cyclictest20785-21grep22:20:251
3108099130,12cyclictest25357-21grep21:10:241
3107699139,3cyclictest25119-21wc22:25:000
3107699139,3cyclictest19421-21sed23:35:200
3107699139,3cyclictest19332-21grep21:00:160
31076991313,0cyclictest0-21swapper/000:25:150
31076991310,2cyclictest8897-21sed19:30:160
31076991310,2cyclictest27516-21sed00:35:200
3107699130,8cyclictest2177-21awk20:25:000
3107699130,8cyclictest11773-21munin-run23:55:000
3108699129,2cyclictest7276-21grep22:35:223
3108699129,2cyclictest32677-21grep22:55:173
3108699129,2cyclictest32039-21head19:10:163
3108699129,2cyclictest32039-21head19:10:153
3108699129,2cyclictest25336-21grep20:05:153
3108699129,2cyclictest19701-21sed22:45:183
3108699129,2cyclictest17946-21wc21:30:013
3108699129,2cyclictest17072-21grep00:25:193
3108699129,2cyclictest13440-21cut19:40:143
3108699129,2cyclictest10599-21tr00:20:153
3108699128,2cyclictest8263-21grep20:35:183
31086991210,1cyclictest7364-21grep21:45:263
31086991210,1cyclictest15399-21grep20:50:183
3108399129,2cyclictest6702-21fschecks_time19:25:182
3108399129,2cyclictest3880-21grep19:20:112
3108399129,2cyclictest31695-21cut21:40:152
3108399129,2cyclictest31546-21grep19:10:142
3108399129,2cyclictest31546-21grep19:10:142
3108399129,2cyclictest20752-21sed22:20:252
3108399129,2cyclictest1684-21tail20:20:232
3108399128,3cyclictest18939-21cut22:45:152
3108399128,3cyclictest13812-21sort21:50:302
3108399128,2cyclictest27367-21cron20:10:002
3108399128,1cyclictest2294-21chrt23:46:372
3108399120,11cyclictest0-21swapper/219:45:152
3108099129,2cyclictest6930-21tr23:50:191
3108099129,2cyclictest5622-21grep00:15:251
3108099129,2cyclictest518-21grep20:20:131
3108099129,2cyclictest29754-21dump-pmu-power20:15:001
3108099129,2cyclictest23151-21head21:05:271
3108099129,2cyclictest23151-21head21:05:261
3108099129,2cyclictest19942-21head21:55:241
3108099129,2cyclictest18847-21grep19:50:151
3108099129,2cyclictest16035-21df19:45:171
3108099129,2cyclictest13636-21ssh00:22:111
3108099128,3cyclictest13057-21head21:25:291
3108099128,2cyclictest687-21ssh23:20:271
3108099128,2cyclictest13604-21head23:55:251
3107699129,2cyclictest7999-21grep23:25:330
3107699129,2cyclictest526-21sed19:10:250
3107699129,2cyclictest526-21sed19:10:240
3107699129,2cyclictest23056-21grep21:05:260
3107699129,2cyclictest23056-21grep21:05:260
3107699128,3cyclictest6765-21head19:25:180
3107699128,3cyclictest31636-21ssh22:05:050
3107699128,2cyclictest11068-21grep19:35:160
3107699122,7cyclictest7744-21snmpd22:26:530
3107699122,5cyclictest224-21systemd-journal22:00:000
3107699120,1cyclictest0-21swapper/021:24:100
3107699120,11cyclictest0-21swapper/023:00:140
3107699120,10cyclictest11602-21ssh21:50:000
3108699118,2cyclictest9561-21ntpq19:30:223
3108699118,2cyclictest8781-21ssh22:11:343
3108699118,2cyclictest6901-21tail23:25:213
3108699118,2cyclictest6892-21cron20:35:003
3108699118,2cyclictest4551-21cut00:15:183
3108699118,2cyclictest4440-21head19:20:133
3108699118,2cyclictest3745-21fgrep20:25:243
3108699118,2cyclictest29571-21chrt20:12:523
3108699118,2cyclictest28511-21ssh00:35:313
3108699118,2cyclictest27333-21grep22:25:283
3108699118,2cyclictest27231-21ssh21:11:503
3108699118,2cyclictest22507-21ssh00:02:433
3108699118,2cyclictest22382-21ssh22:22:113
3108699118,2cyclictest19407-21cat21:00:163
3108699118,2cyclictest19023-21chrt21:30:173
3108699118,2cyclictest18941-21ls21:55:173
3108699118,2cyclictest16532-21sh23:07:483
3108699118,2cyclictest13034-21ssh22:15:183
3108699118,2cyclictest10432-21rm23:03:163
3108699111,2cyclictest224-21systemd-journal19:50:113
3108399119,1cyclictest19366-21grep21:55:192
3108399118,2cyclictest6911-21wc22:10:192
3108399118,2cyclictest555-21tail00:10:332
3108399118,2cyclictest28873-21grep20:10:232
3108399118,2cyclictest25267-21sh23:40:152
3108399118,2cyclictest1992-21grep19:15:182
3108399118,2cyclictest15493-21ls20:50:202
3108399118,2cyclictest14158-21grep23:30:292
3108399118,2cyclictest13870-21fw_packets19:40:162
3108399118,1cyclictest29144-21kworker/2:123:55:012
3108399117,3cyclictest13446-21wc20:45:282
3108399116,4cyclictest19024-21diskmemload21:30:182
3108399115,3cyclictest23554-21diskmemload22:00:122
3108099118,2cyclictest7817-21ssh21:45:491
3108099118,2cyclictest4980-21ssh21:44:511
3108099118,2cyclictest32145-21taskset20:19:371
3108099118,2cyclictest29693-21ssh21:38:361
3108099118,2cyclictest28914-21ssh00:36:041
3108099118,2cyclictest28020-21ssh00:06:521
3108099118,2cyclictest25937-21grep20:05:181
3108099118,2cyclictest23647-21sed20:00:161
3108099118,2cyclictest21825-21ssh00:01:501
3108099118,2cyclictest16889-21sh00:25:181
3108099118,2cyclictest14919-21cut20:50:151
3108099117,3cyclictest12181-21grep20:45:171
3108099117,2cyclictest1944-21sort19:15:171
3107699119,1cyclictest18859-21grep19:50:150
3107699118,2cyclictest9401-21wc20:40:000
3107699118,2cyclictest31975-21ssh22:30:130
3107699118,2cyclictest27382-21dump-pmu-power20:09:590
3107699118,2cyclictest26167-21ssh23:40:190
3107699118,2cyclictest22808-21grep00:30:240
3107699118,2cyclictest20873-21grep23:10:270
3107699118,2cyclictest1991-21sh21:17:090
3107699118,2cyclictest19514-21grep22:45:170
3107699118,2cyclictest17401-21grep20:55:190
3107699118,2cyclictest14681-21rm23:05:480
3107699118,2cyclictest1421-21ssh00:11:450
3107699118,2cyclictest12522-21tr22:15:150
3107699117,3cyclictest28369-21tail20:10:190
3107699115,2cyclictest224-21systemd-journal00:09:380
3107699110,10cyclictest535-21gzip21:40:230
3107699110,10cyclictest27454-21ssh21:36:090
3107699110,10cyclictest27383-21ssh23:15:360
3108699108,1cyclictest6094-21aten2_r7power_e19:25:153
3108699108,1cyclictest32431-21taskset20:20:033
3108699108,1cyclictest29987-21cat20:15:103
3108699108,1cyclictest213962sleep321:05:013
3108699108,1cyclictest213962sleep321:05:013
3108699108,1cyclictest15873-21expr19:45:163
3108699107,2cyclictest22877-21basename20:00:103
3108699100,9cyclictest20876-21cat19:55:153
3108399108,1cyclictest7953-21chrt20:35:152
3108399108,1cyclictest6064-21basename23:25:152
3108399108,1cyclictest3393-21cat20:25:202
3108399108,1cyclictest199662sleep219:51:402
3108399108,1cyclictest19213-21egrep21:00:162
3108399107,2cyclictest9590-21grep20:40:132
3108399107,2cyclictest11755-21sed19:35:192
3108099108,1cyclictest31755-21cat19:10:141
3108099108,1cyclictest31755-21cat19:10:141
3108099108,1cyclictest14268-21grep19:40:191
3108099107,2cyclictest7441-21tr20:35:131
3108099107,2cyclictest5020-21tr20:30:151
3108099107,2cyclictest18703-21ssh23:35:071
3108099107,2cyclictest11181-21sed20:40:251
3108099100,9cyclictest12360-21rm23:05:081
3107699108,1cyclictest3041-21grep20:25:180
3107699108,1cyclictest10020-21expr20:40:140
3107699107,2cyclictest24864-21taskset20:03:250
3107699107,2cyclictest13932-21head19:40:160
3107699102,4cyclictest224-21systemd-journal20:55:000
3107699100,9cyclictest30551-21df20:15:150
3107699100,9cyclictest0-21swapper/019:15:140
310839996,2cyclictest30560-21sed20:15:142
310769997,1cyclictest12240-21cat20:45:160
310769995,2cyclictest4866-21grep20:30:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional