You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-29 - 05:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Wed Jul 29, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1428229482,8sleep00-21swapper/019:07:460
1411728966,6sleep10-21swapper/119:05:391
1415828370,9sleep30-21swapper/319:06:093
1231928169,8sleep20-21swapper/219:05:042
238922590,0sleep00-21swapper/000:32:590
1455399321,26cyclictest21013-21awk22:15:223
1455399311,29cyclictest8703-21ls00:05:003
265532300,1sleep2311rcuc/223:55:272
1455399300,1cyclictest0-21swapper/322:45:153
1455399290,28cyclictest26526-21dump-pmu-power22:00:003
1455399290,28cyclictest0-21swapper/300:38:203
1455399290,21cyclictest0-21swapper/321:27:003
1455399290,1cyclictest0-21swapper/323:24:023
1455399290,0cyclictest0-21swapper/323:46:073
14545992928,1cyclictest0-21swapper/021:26:260
1455399282,25cyclictest0-21swapper/300:17:193
1455399280,27cyclictest0-21swapper/323:36:333
1455399280,27cyclictest0-21swapper/320:15:143
1455399280,25cyclictest0-21swapper/321:12:593
1455399280,1cyclictest7564-21diskmemload00:34:563
1455399280,1cyclictest0-21swapper/322:36:553
1455399271,25cyclictest7564-21diskmemload22:26:573
1455399270,4cyclictest7564-21diskmemload22:50:083
1455399270,3cyclictest7564-21diskmemload22:56:333
14551992726,1cyclictest0-21swapper/219:26:182
1454799273,4cyclictest7564-21diskmemload21:10:151
14545992723,4cyclictest13-21ksoftirqd/023:31:330
1455399261,24cyclictest7564-21diskmemload22:42:223
1455399261,16cyclictest0-21swapper/323:25:203
1455399260,25cyclictest0-21swapper/319:40:133
1455399260,24cyclictest0-21swapper/323:33:373
1455399260,24cyclictest0-21swapper/320:50:153
1455399260,24cyclictest0-21swapper/320:45:173
1455399260,1cyclictest0-21swapper/300:20:243
1455399254,20cyclictest7564-21diskmemload21:21:193
1455399254,20cyclictest7564-21diskmemload21:21:183
1455399252,22cyclictest7564-21diskmemload22:09:053
1455399243,19cyclictest17782-21ssh00:10:123
14553992421,2cyclictest17086-21df_inode21:15:193
1455399241,22cyclictest0-21swapper/320:25:143
1455399240,23cyclictest0-21swapper/323:19:153
1455399240,1cyclictest0-21swapper/322:30:143
14551992421,2cyclictest26038-21apache_accesses00:15:132
1455199240,23cyclictest0-21swapper/223:24:412
1454799240,22cyclictest0-21swapper/122:32:281
14545992421,2cyclictest8108-21apache_processe23:45:150
14545992421,2cyclictest22864-21apache_accesses23:15:120
14553992320,2cyclictest7400-21memory23:05:233
14553992320,2cyclictest17575-21apache_accesses21:35:123
1455399231,19cyclictest28601-21grep19:35:263
1455399230,22cyclictest21101-21ssh23:52:353
1455399230,1cyclictest0-21swapper/300:06:393
1455399230,19cyclictest0-21swapper/322:23:183
1455399230,17cyclictest0-21swapper/322:10:353
1455399230,0cyclictest0-21swapper/323:43:083
14551992320,2cyclictest8497-21ntp_states23:25:202
14551992320,2cyclictest11295-21apache_volume22:10:132
1455199230,15cyclictest7564-21diskmemload23:30:212
14545992320,2cyclictest32740-21df_inode23:40:140
1454599231,1cyclictest21990-21ssh23:53:170
1455399222,19cyclictest7564-21diskmemload23:55:193
14553992220,1cyclictest27052-21apache_volume22:00:123
1455399221,21cyclictest7564-21diskmemload23:02:323
1455399220,2cyclictest18276-21cut21:55:003
1455399220,21cyclictest7564-21diskmemload21:42:173
1455399220,21cyclictest4739-21grep21:00:253
1455399220,20cyclictest0-21swapper/320:00:103
14551992220,1cyclictest3985-21apache_processe22:25:102
14551992220,1cyclictest30775-21diskstats23:00:182
14551992220,1cyclictest23827-21apache_accesses23:35:122
14551992220,1cyclictest22851-21memory22:55:212
14551992220,1cyclictest10288-21apache_volume00:25:132
14551992219,2cyclictest696-21apache_processe21:25:132
14551992219,2cyclictest32106-21apache_accesses23:40:112
14551992219,2cyclictest18232-21df_inode21:35:162
14547992218,1cyclictest0-21swapper/122:54:221
1454799220,4cyclictest0-21swapper/100:31:511
14545992220,1cyclictest3972-21apache_accesses22:25:100
14545992219,2cyclictest32619-21ntp_states23:20:230
14545992219,2cyclictest2960-21apache_processe22:05:110
14545992219,2cyclictest28086-21irqstats22:00:200
14545992219,2cyclictest13334-21apache_accesses22:50:110
1454599221,1cyclictest141rcu_preempt23:02:430
1455399212,14cyclictest0-21swapper/320:40:263
1455399211,19cyclictest7564-21diskmemload00:26:543
1455399211,19cyclictest0-21swapper/321:45:283
1455399211,17cyclictest15380-21ssh23:10:173
1455399210,3cyclictest22193-21aten2_r7power_e19:25:133
1455399210,17cyclictest0-21swapper/321:31:043
14551992119,1cyclictest19414-21perl00:30:172
14551992118,2cyclictest17343-21hddtemp_smartct21:15:192
14551992118,2cyclictest13349-21apache_volume22:50:122
14547992114,6cyclictest18653-21apache_processe21:55:121
14545992118,2cyclictest4528-21munin-run22:45:000
1455399202,17cyclictest4226-21/usr/sbin/munin19:55:213
1455399201,15cyclictest17162-21apt-get19:15:153
14551992019,1cyclictest0-21swapper/221:54:552
14551992017,2cyclictest8732-21diskstats21:10:172
14551992017,2cyclictest27700-21diskstats22:00:162
14551992017,2cyclictest25458-21df_inode21:20:162
14551992017,2cyclictest25458-21df_inode21:20:152
14551992017,2cyclictest18308-21diskstats00:10:152
14551992017,2cyclictest10798-21munin-run22:09:592
1455199200,19cyclictest0-21swapper/221:44:312
14547992015,4cyclictest0-21swapper/122:40:131
14547992014,5cyclictest15518-21apache_accesses23:30:131
14547992012,2cyclictest12622-21ntp_states22:10:211
1454599205,8cyclictest2818-21awk21:00:000
1455399197,9cyclictest1149-21taskset19:46:543
14551991917,1cyclictest10281-21irqstats00:05:212
14547991917,1cyclictest14606-21aten2_r7power_e23:10:141
14547991914,4cyclictest3134-21memory21:45:201
14547991913,5cyclictest25791-21irqstats23:55:191
14545991916,2cyclictest13309-21users22:10:270
1455399181,16cyclictest16100-21find19:10:263
14551991817,1cyclictest0-21swapper/222:41:312
14551991816,1cyclictest26971-21apache_processe00:35:122
1454799184,5cyclictest225-21systemd-journal19:30:011
1454799182,3cyclictest1458-21latency_hist21:45:011
14547991811,2cyclictest16434-21grep19:10:301
1454799180,4cyclictest0-21swapper/121:24:461
1454799180,4cyclictest0-21swapper/121:24:451
1454599185,6cyclictest8707-21munin-run00:05:010
1455399171,15cyclictest3235-21timerwakeupswit19:50:263
14551991714,2cyclictest18640-21apache_accesses21:55:132
1455199170,2cyclictest0-21swapper/200:21:042
1454799179,7cyclictest10550irq/122-ahci[0022:25:231
1454799179,2cyclictest27300-21tail00:15:211
1454799179,2cyclictest16494-21cut23:50:151
14547991713,3cyclictest7564-21diskmemload00:03:151
14547991710,2cyclictest9513-21sed23:45:251
1454799170,16cyclictest0-21swapper/119:40:131
1455399168,7cyclictest2364-21sed20:55:293
1455399161,14cyclictest24673-21rt-features20:35:213
1455399160,15cyclictest21604-21fschecks_time20:30:173
1455399160,15cyclictest19801-21wget19:20:143
1455399160,15cyclictest19801-21wget19:20:133
14551991616,0cyclictest0-21swapper/222:32:522
1454799169,2cyclictest25865-21grep20:40:141
1454799169,2cyclictest2352-21sed20:55:281
1454799168,3cyclictest22647-21grep20:30:261
14547991614,1cyclictest6661-21sensors_fan22:45:251
14547991614,1cyclictest29438-21ntp_states22:20:221
14547991614,1cyclictest21695-21apache_accesses22:55:141
14547991614,1cyclictest1947-21apache_accesses00:20:131
14547991614,1cyclictest17587-21apache_processe21:35:121
14547991614,1cyclictest10351-21apache_volume21:50:151
14547991614,1cyclictest0-21swapper/123:05:001
14547991613,2cyclictest18986-21ntp_states00:10:201
1454599164,7cyclictest21899-21perf19:25:010
1454599164,7cyclictest21899-21perf19:25:000
1454599162,8cyclictest12881-21munin-run22:50:010
14545991613,2cyclictest11600-21ntp_states00:25:220
1455399158,6cyclictest16494-21expr20:20:143
1455199159,2cyclictest24180-21ntpq23:15:232
14551991513,1cyclictest0-21swapper/221:47:412
1455199150,15cyclictest0-21swapper/223:46:332
1454799158,2cyclictest10926-21grep21:30:281
1454799150,2cyclictest0-21swapper/122:08:141
1454599154,6cyclictest6407-21basename20:00:000
14545991512,2cyclictest26977-21apache_volume00:35:120
1455399147,6cyclictest5697-21expr21:05:153
1455399141,12cyclictest26602-21taskset19:34:053
14551991410,2cyclictest1415-21sleep19:50:012
1455199140,1cyclictest0-21swapper/222:38:142
1454599146,4cyclictest0-21swapper/019:20:010
14545991414,0cyclictest0-21swapper/022:31:250
14545991411,3cyclictest13-21ksoftirqd/021:55:330
14545991411,2cyclictest23788-21tail19:25:270
14545991410,3cyclictest17807-21sed20:20:240
1454599140,13cyclictest0-21swapper/022:23:220
1454599140,13cyclictest0-21swapper/021:50:160
143622140,0chrt0-21swapper/021:33:030
1455399136,6cyclictest11253-21cut20:10:003
1455399130,0cyclictest0-21swapper/320:10:143
14551991310,2cyclictest9785-21sed20:05:162
14551991310,2cyclictest29147-21cat19:40:012
14551991310,2cyclictest26310-21grep20:40:152
1455199130,12cyclictest0-21swapper/219:50:142
14547991310,2cyclictest19884-21basename19:20:141
14547991310,2cyclictest19884-21basename19:20:141
14547991310,2cyclictest12620-21sed20:10:221
1454799130,2cyclictest0-21swapper/100:06:541
1454799130,13cyclictest0-21swapper/121:17:311
1454599139,3cyclictest31455-21grep20:50:200
1454599139,2cyclictest0-21swapper/023:27:290
1454599136,6cyclictest29571-21munin-run23:00:010
14545991311,1cyclictest13-21ksoftirqd/000:06:380
14545991310,2cyclictest9771-21munin-run00:25:000
14545991310,2cyclictest5711-21grep21:05:140
14545991310,2cyclictest14426-21cut20:15:170
14545991310,2cyclictest13983-21cron23:10:000
1455199129,2cyclictest8754-21cron20:05:002
1455199129,2cyclictest7061-21grep23:05:202
1455199129,2cyclictest28806-21grep22:20:172
1455199129,2cyclictest20303-21grep22:15:172
1455199129,2cyclictest1173-21tr00:00:152
1455199127,4cyclictest520-21gmain20:27:512
1454799129,2cyclictest7146-21grep21:05:261
1454799129,2cyclictest6367-21sh20:00:001
1454799129,2cyclictest3776-21grep21:00:171
1454799129,2cyclictest32456-21tail19:45:181
1454799129,2cyclictest29236-21grep20:45:211
1454799129,2cyclictest25770-21sed19:30:201
1454799129,2cyclictest2531-21sort21:25:261
1454799129,2cyclictest24725-21grep23:35:171
1454799129,2cyclictest23679-21grep23:15:181
1454799129,2cyclictest17776-21cut19:15:181
1454799129,2cyclictest1644-21grep23:40:261
1454799128,3cyclictest28563-21cron22:40:011
1454799128,3cyclictest25498-21systemctl20:40:001
1454599129,2cyclictest27999-21grep00:15:290
1454599129,2cyclictest25502-21sort21:20:170
1454599129,2cyclictest25502-21sort21:20:160
1454599129,2cyclictest24733-21apache_processe23:55:130
1454599129,2cyclictest18468-21tail00:10:160
1454599129,2cyclictest13103-21grep20:10:270
1454599128,3cyclictest23505-21munin-node20:30:280
1454599128,3cyclictest15425-21grep19:10:200
14545991210,1cyclictest13-21ksoftirqd/021:36:240
51252110,0sleep10-21swapper/123:23:491
1455199119,1cyclictest10339-21grep21:30:212
1455199118,2cyclictest3958-21cat21:00:172
1455199118,2cyclictest19234-21ssh23:51:072
1455199118,2cyclictest17135-21basename19:15:132
1455199118,2cyclictest16900-21tail20:20:162
1455199118,2cyclictest16333-21expr23:10:262
1455199118,2cyclictest14917-21tr20:15:222
1455199118,2cyclictest13245-21taskset20:11:442
1455199110,1cyclictest0-21swapper/222:45:302
1454799119,1cyclictest21116-21grep22:15:231
1454799118,2cyclictest9799-21sed20:05:161
1454799118,2cyclictest951-21ssh22:03:501
1454799118,2cyclictest8680-21chrt20:04:051
1454799118,2cyclictest8448-21awk23:25:191
1454799118,2cyclictest3820-21sed19:55:011
1454799118,2cyclictest27275-21tr00:35:151
1454799118,2cyclictest14049-21tr20:15:151
1454799110,2cyclictest0-21swapper/123:05:041
1454599118,2cyclictest917-21sed19:45:280
1454599118,2cyclictest8476-21taskset20:01:280
1454599118,2cyclictest5840-21ssh21:46:530
1454599118,2cyclictest30978-21grep19:40:260
1454599118,2cyclictest28913-21tail20:45:180
1454599118,2cyclictest24843-21tr19:30:130
1454599118,2cyclictest24519-21sort23:35:170
1454599118,2cyclictest20672-21apt-get22:35:120
1454599118,2cyclictest20301-21df22:15:170
1454599118,2cyclictest17179-21ssh21:15:190
1454599118,2cyclictest15190-21cut23:10:160
1454599118,2cyclictest14426-21rm21:13:520
1454599117,3cyclictest27424-21tail20:40:260
1454599112,6cyclictest4612-21timerwakeupswit21:00:260
1454599110,3cyclictest0-21swapper/021:43:150
1455199108,1cyclictest973-21cat20:55:172
1455199108,1cyclictest5783-21expr21:05:152
1455199108,1cyclictest31817-21cat20:50:222
1455199108,1cyclictest30403-21cat19:40:192
1455199108,1cyclictest28625-21sed20:45:172
1455199108,1cyclictest26059-21expr19:30:242
1455199108,1cyclictest23494-21egrep20:35:152
1455199108,1cyclictest21041-21awk20:30:152
1455199108,1cyclictest20181-21expr19:20:152
1455199108,1cyclictest20181-21expr19:20:142
1455199108,1cyclictest16845-21taskset19:14:402
1455199107,2cyclictest4500-21tr19:55:142
1454799108,1cyclictest32299-21sed20:50:281
1454799108,1cyclictest13442-21sh00:26:211
1454799107,2cyclictest28869-21chrt19:37:441
1454799107,2cyclictest19483-21grep20:25:201
1454799106,2cyclictest16410-21tr20:20:141
1454599108,1cyclictest2435-21sed19:50:180
145459997,1cyclictest9098-21wget20:05:130
145459997,1cyclictest27537-21taskset19:35:150
145459997,1cyclictest20305-21chrt20:25:320
145459996,2cyclictest24526-21sed20:35:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional