You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-21 - 06:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot3s.osadl.org (updated Tue Jul 21, 2026 00:45:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1402139998541,27cyclictest16850irq/134-800000021:40:010
140213999851,80cyclictest1437411-21ssh22:11:110
140213999832,76cyclictest1409383-21ssh21:11:510
1402139998124,52cyclictest1502829-21ssh00:28:080
1402139998027,49cyclictest1426785-21ssh21:48:340
140213999802,73cyclictest1471504-21ssh23:22:390
1402139998025,50cyclictest1474188-21ssh23:28:240
1402139998024,39cyclictest1453290-21ssh22:44:420
1402139998024,39cyclictest1453290-21ssh22:44:420
1402139998021,54cyclictest1476897-21ssh23:34:010
1402139998016,14cyclictest1480183-21ssh23:40:270
1402139997926,36cyclictest1491317-21ssh00:04:140
1402139997924,51cyclictest1505400-21ssh00:33:350
1402139997834,26cyclictest1416843-21ssh21:27:320
1402139997833,28cyclictest1431435-21ssh21:58:130
140213999783,21cyclictest16850irq/134-800000023:20:020
140213999782,13cyclictest48-21kcompactd023:45:320
140213999771,59cyclictest1493648-21ssh00:09:090
1402139997634,25cyclictest1427974-21ssh21:50:420
1402139997616,14cyclictest1441946-21ssh22:20:310
1402139997615,44cyclictest16850irq/134-800000021:44:310
1402139997528,13cyclictest1413148-21tr21:19:590
140213999751,70cyclictest0-21swapper/022:57:360
1402139997424,32cyclictest1434210-21ssh22:04:070
1402139997419,50cyclictest1488714-21ssh23:58:530
1402139997317,38cyclictest1464370-21ssh23:07:460
140214799722,46cyclictest291rcuc/220:05:012
1402139997227,26cyclictest1435502-21ssh22:06:400
1402139997225,31cyclictest1496154-21ssh00:14:180
1402139997225,30cyclictest1453808-21ssh22:45:260
1402139997127,28cyclictest1478403-21ssh23:37:010
140213999712,14cyclictest1408310-21sshd21:09:150
140213999711,65cyclictest0-21swapper/022:39:070
140213999701,53cyclictest0-21swapper/000:19:340
140213999692,63cyclictest1448262-21ssh22:34:040
1402139996918,33cyclictest1446209-21ssh22:29:580
140213999671,16cyclictest1457038-21ssh22:52:310
140213999671,16cyclictest1457038-21ssh22:52:310
1402139996624,24cyclictest0-21swapper/000:35:110
1402139996617,33cyclictest1467525-21ssh23:14:320
1402139996521,27cyclictest1461996-21ssh23:02:400
1402139996518,42cyclictest1441253-21ssh22:19:280
140213999642,13cyclictest1404388-21sshd19:50:140
140215299633,4cyclictest131rcu_preempt23:50:033
140213999632,13cyclictest1408076-21munin-run21:05:000
140213999631,15cyclictest1405762-21munin-run20:20:020
140213999625,53cyclictest1501507-21date00:25:030
140213999622,16cyclictest1485273-21ssh23:51:190
140213999621,57cyclictest0-21swapper/021:34:510
140213999621,14cyclictest1406729-21sshd20:37:150
140213999611,4cyclictest0-21swapper/021:22:560
140213999602,13cyclictest1403150-21munin-run19:30:020
140213999602,13cyclictest1403150-21munin-run19:30:010
140213999602,12cyclictest3467602-21systemd-journal20:05:010
140213999601,13cyclictest48-21kcompactd019:30:180
140213999601,13cyclictest48-21kcompactd019:30:170
140214299598,8cyclictest23-21ksoftirqd/122:15:011
140214299591,8cyclictest23-21ksoftirqd/123:45:011
140213999592,9cyclictest1403627-21cron19:40:000
1402139995915,9cyclictest309-21in:imuxsock19:17:000
1402139995913,8cyclictest1406012-21systemd-run20:25:010
1402147995749,5cyclictest1441573-21kworker/2:323:50:022
1402142995730,22cyclictest221rcuc/119:35:021
1402142995730,22cyclictest221rcuc/119:35:011
140213999573,14cyclictest1407823-21modprobe21:00:010
1402139995715,38cyclictest1402860-21sshd19:22:090
140214299566,9cyclictest23-21ksoftirqd/123:52:481
140214299564,10cyclictest23-21ksoftirqd/121:00:001
140213999562,11cyclictest1404642-21modprobe19:55:160
1402147995546,5cyclictest1391255-21kworker/2:020:20:012
1402147995544,7cyclictest1391255-21kworker/2:019:25:012
140214799551,16cyclictest1460535-21modprobe22:59:592
140213999552,16cyclictest1402227-21latency_hist19:10:030
140213999552,16cyclictest1402227-21latency_hist19:10:020
1402139995517,34cyclictest1406234-21kworker/u8:120:30:000
1402139995515,35cyclictest309-21in:imuxsock20:35:000
140213999551,12cyclictest364-21snmpd19:46:120
1402147995444,6cyclictest1391255-21kworker/2:021:40:022
140214299547,10cyclictest23-21ksoftirqd/121:36:521
140214299543,12cyclictest23-21ksoftirqd/100:38:051
1402142995427,16cyclictest3467602-21systemd-journal21:45:011
140213999541,14cyclictest364-21snmpd20:54:250
140215299537,6cyclictest131rcu_preempt22:25:003
140214799539,10cyclictest131rcu_preempt22:00:002
1402147995345,5cyclictest1441573-21kworker/2:322:35:092
1402147995344,5cyclictest1441573-21kworker/2:323:29:222
140213999535,7cyclictest1407309-21latency_hist20:50:010
140213999531,5cyclictest1407038-21munin-run20:45:010
140213999531,48cyclictest0-21swapper/020:15:030
1402152995219,23cyclictest37-21ksoftirqd/319:20:023
1402147995244,5cyclictest1391255-21kworker/2:022:25:002
1402147995244,5cyclictest1391255-21kworker/2:022:02:472
1402147995244,5cyclictest1391255-21kworker/2:021:45:022
1402147995243,6cyclictest1441573-21kworker/2:323:45:012
1402147995243,5cyclictest1441573-21kworker/2:300:40:022
1402147995243,5cyclictest1391255-21kworker/2:019:40:022
1402147995242,7cyclictest1391255-21kworker/2:022:10:562
1402142995211,5cyclictest131rcu_preempt23:10:171
140213999521,14cyclictest1403931-21sh19:45:000
140213999521,14cyclictest1403931-21sh19:45:000
1402147995143,5cyclictest1441573-21kworker/2:323:35:032
1402147995143,5cyclictest1391255-21kworker/2:021:16:472
1402147995142,5cyclictest1441573-21kworker/2:322:40:452
1402147995142,5cyclictest1441573-21kworker/2:322:40:452
1402147995141,6cyclictest1391255-21kworker/2:023:06:102
1402147995132,15cyclictest1408511-21cat21:10:022
1402147995130,15cyclictest275-21dbus-daemon23:25:012
140214299518,8cyclictest23-21ksoftirqd/123:26:211
140214299516,8cyclictest23-21ksoftirqd/100:25:011
140214299511,17cyclictest0-21swapper/122:35:101
1402142995110,6cyclictest131rcu_preempt23:50:031
1402147995042,5cyclictest1441573-21kworker/2:300:25:502
1402147995042,5cyclictest1391255-21kworker/2:021:10:322
1402147995042,5cyclictest1391255-21kworker/2:000:23:242
1402147995041,5cyclictest1441573-21kworker/2:323:12:542
1402147995041,5cyclictest1441573-21kworker/2:300:34:072
1402147995036,11cyclictest1-21systemd20:30:012
1402147995034,12cyclictest3467602-21systemd-journal19:15:002
1402147995027,7cyclictest1391255-21kworker/2:000:01:532
140214299508,6cyclictest131rcu_preempt22:35:011
140214299507,9cyclictest23-21ksoftirqd/121:50:271
140214299504,5cyclictest131rcu_preempt00:15:041
1402142995038,9cyclictest23-21ksoftirqd/121:28:191
140214299501,8cyclictest1463013-21ssh23:05:011
1402152994931,13cyclictest37-21ksoftirqd/320:50:013
1402152994914,6cyclictest131rcu_preempt21:35:023
1402147994942,5cyclictest1391255-21kworker/2:022:47:122
1402147994942,4cyclictest1441573-21kworker/2:322:32:002
1402147994941,5cyclictest1441573-21kworker/2:323:15:102
1402147994941,5cyclictest1391255-21kworker/2:022:15:432
1402147994941,5cyclictest1391255-21kworker/2:000:13:042
1402147994941,4cyclictest1391255-21kworker/2:019:50:002
1402147994940,6cyclictest1441573-21kworker/2:323:39:512
1402147994940,5cyclictest1391255-21kworker/2:022:51:042
1402147994940,5cyclictest1391255-21kworker/2:022:51:042
1402147994939,6cyclictest1391255-21kworker/2:021:32:012
140214299497,6cyclictest131rcu_preempt22:45:341
140214299497,11cyclictest23-21ksoftirqd/100:00:011
140214299496,10cyclictest23-21ksoftirqd/123:21:111
1402142994940,6cyclictest23-21ksoftirqd/123:39:481
1402142994940,6cyclictest23-21ksoftirqd/121:23:571
1402142994933,12cyclictest3467602-21systemd-journal19:55:011
140214299493,12cyclictest23-21ksoftirqd/122:10:011
1402142994912,5cyclictest131rcu_preempt23:15:171
140214299491,18cyclictest0-21swapper/100:11:101
1402142994910,6cyclictest131rcu_preempt00:07:081
140215299488,6cyclictest131rcu_preempt00:00:033
140215299485,34cyclictest37-21ksoftirqd/322:50:023
1402147994841,5cyclictest1441573-21kworker/2:323:53:152
1402147994841,3cyclictest1391255-21kworker/2:021:48:482
1402147994840,5cyclictest1391255-21kworker/2:023:01:522
1402147994840,5cyclictest1391255-21kworker/2:000:18:032
1402147994839,5cyclictest1391255-21kworker/2:020:25:022
140214299488,8cyclictest23-21ksoftirqd/122:25:001
1402142994811,5cyclictest131rcu_preempt23:05:291
1402142994810,7cyclictest131rcu_preempt22:57:551
1402142994810,5cyclictest131rcu_preempt00:05:021
1402152994731,12cyclictest275-21dbus-daemon23:15:023
1402152994721,4cyclictest37-21ksoftirqd/323:30:023
1402147994739,5cyclictest1417928-21kworker/2:221:51:132
1402147994739,5cyclictest1391255-21kworker/2:022:07:152
1402147994739,5cyclictest1391255-21kworker/2:000:05:402
1402147994738,6cyclictest1391255-21kworker/2:021:20:332
140214299478,12cyclictest23-21ksoftirqd/121:46:081
140214299474,11cyclictest23-21ksoftirqd/120:45:001
1402142994732,11cyclictest309-21in:imuxsock19:40:001
140214299472,13cyclictest23-21ksoftirqd/100:31:181
1402142994714,14cyclictest23-21ksoftirqd/120:55:021
1402142994713,8cyclictest23-21ksoftirqd/121:31:081
1402142994713,10cyclictest23-21ksoftirqd/121:10:351
140214299471,19cyclictest0-21swapper/121:16:421
1402142994711,18cyclictest131rcu_preempt22:19:201
140213999471,4cyclictest0-21swapper/019:12:390
140215299464,3cyclictest131rcu_preempt23:53:113
1402152994636,5cyclictest311-21rs:main3
140215299463,4cyclictest131rcu_preempt23:16:173
140215299463,3cyclictest131rcu_preempt23:42:513
1402147994639,5cyclictest1391255-21kworker/2:022:29:052
140214799463,5cyclictest131rcu_preempt00:00:002
1402147994628,14cyclictest1-21systemd20:55:022
140214299468,5cyclictest131rcu_preempt20:20:021
140214299463,8cyclictest23-21ksoftirqd/122:54:421
140214299463,8cyclictest23-21ksoftirqd/122:54:411
1402142994633,9cyclictest23-21ksoftirqd/122:01:571
140214299463,12cyclictest23-21ksoftirqd/122:30:021
1402142994628,13cyclictest3467602-21systemd-journal19:15:001
1402142994613,6cyclictest131rcu_preempt23:31:361
1402142994613,6cyclictest131rcu_preempt21:05:021
1402142994610,5cyclictest131rcu_preempt20:04:141
140215299456,5cyclictest131rcu_preempt00:38:553
140215299456,21cyclictest0-21swapper/300:27:103
140215299455,3cyclictest131rcu_preempt21:26:473
140215299454,3cyclictest131rcu_preempt22:11:553
140215299453,3cyclictest150-21pr/ttyS223:00:003
1402152994528,13cyclictest3467602-21systemd-journal21:10:153
140215299451,4cyclictest131rcu_preempt22:44:153
140215299451,4cyclictest131rcu_preempt22:44:143
1402147994536,6cyclictest1391255-21kworker/2:020:15:022
140214299457,5cyclictest131rcu_preempt21:57:401
1402142994527,13cyclictest1404159-21cron19:49:591
1402142994514,6cyclictest131rcu_preempt19:10:031
1402142994514,6cyclictest131rcu_preempt19:10:021
1402142994513,7cyclictest131rcu_preempt00:25:121
140214299451,14cyclictest0-21swapper/122:42:341
140214299451,14cyclictest0-21swapper/122:42:331
140213999451,15cyclictest0-21swapper/020:06:110
140215299444,4cyclictest131rcu_preempt22:02:313
1402152994430,11cyclictest37-21ksoftirqd/300:21:533
1402152994428,13cyclictest37-21ksoftirqd/322:09:593
1402152994425,15cyclictest3467602-21systemd-journal23:07:153
140215299441,4cyclictest131rcu_preempt20:13:143
140214799447,5cyclictest131rcu_preempt20:40:002
1402147994436,5cyclictest1391255-21kworker/2:021:28:332
1402147994436,5cyclictest1391255-21kworker/2:020:10:012
1402147994429,10cyclictest309-21in:imuxsock19:35:002
1402147994429,10cyclictest309-21in:imuxsock19:35:002
1402142994435,6cyclictest23-21ksoftirqd/120:15:011
1402142994433,8cyclictest23-21ksoftirqd/119:30:021
1402142994433,8cyclictest23-21ksoftirqd/119:30:011
140214299443,10cyclictest23-21ksoftirqd/121:10:001
140215299435,3cyclictest131rcu_preempt22:35:353
140215299434,3cyclictest131rcu_preempt22:32:593
1402152994331,9cyclictest3467602-21systemd-journal21:00:033
1402152994331,9cyclictest1403897-21sshd19:42:163
1402152994331,9cyclictest1403897-21sshd19:42:153
1402152994329,11cyclictest37-21ksoftirqd/300:04:563
1402152994328,12cyclictest37-21ksoftirqd/323:38:443
1402152994327,11cyclictest275-21dbus-daemon21:40:013
140215299432,3cyclictest131rcu_preempt22:53:253
140215299432,3cyclictest131rcu_preempt22:53:243
1402152994321,4cyclictest37-21ksoftirqd/323:23:533
140214299439,5cyclictest131rcu_preempt20:06:161
1402152994226,13cyclictest37-21ksoftirqd/321:00:003
1402152994226,12cyclictest3467602-21systemd-journal20:51:153
1402142994230,7cyclictest309-21in:imuxsock19:25:001
1402142994228,10cyclictest3467602-21systemd-journal20:35:141
1402142994228,10cyclictest1-21systemd20:30:001
140215299413,6cyclictest131rcu_preempt21:57:393
140215299413,4cyclictest131rcu_preempt21:09:593
1402152994128,10cyclictest1-21systemd19:25:013
1402152994126,11cyclictest37-21ksoftirqd/321:51:303
140215299412,3cyclictest131rcu_preempt23:04:463
140215299412,35cyclictest37-21ksoftirqd/321:45:033
1402152994116,6cyclictest37-21ksoftirqd/300:19:563
140215299411,5cyclictest131rcu_preempt21:23:193
140215299411,4cyclictest131rcu_preempt21:15:293
1402147994132,6cyclictest1391255-21kworker/2:019:20:022
1402147994128,9cyclictest1407302-21systemd-run20:50:012
140215299405,4cyclictest131rcu_preempt00:32:003
140215299404,5cyclictest131rcu_preempt00:10:023
1402152994026,11cyclictest37-21ksoftirqd/320:45:013
1402152994024,10cyclictest1-21systemd20:05:013
140215299401,3cyclictest131rcu_preempt23:30:443
140215299401,3cyclictest131rcu_preempt22:20:003
1402147994031,6cyclictest1391255-21kworker/2:020:35:002
1402147994019,4cyclictest30-21ksoftirqd/220:00:012
1402147994014,5cyclictest1391255-21kworker/2:021:00:012
1402152993929,7cyclictest1405983-21sshd20:24:143
140215299392,6cyclictest131rcu_preempt19:40:003
1402152993915,4cyclictest37-21ksoftirqd/322:27:313
140214799392,3cyclictest131rcu_preempt21:05:002
1402142993910,6cyclictest131rcu_preempt20:35:011
1402152993825,11cyclictest37-21ksoftirqd/319:27:153
1402152993825,11cyclictest37-21ksoftirqd/319:27:143
140215299381,3cyclictest131rcu_preempt21:45:313
1402147993825,10cyclictest1407104-21cat20:45:012
1402142993825,10cyclictest23-21ksoftirqd/119:20:021
140214299382,31cyclictest1403904-21sshd19:44:161
140214299382,31cyclictest1403904-21sshd19:44:151
1402142993810,5cyclictest131rcu_preempt20:25:021
1402152993725,10cyclictest37-21ksoftirqd/319:55:013
140214799374,4cyclictest131rcu_preempt19:45:012
140214799374,4cyclictest131rcu_preempt19:45:002
140214799372,8cyclictest0-21swapper/219:25:202
140214799372,8cyclictest0-21swapper/219:25:192
140214299371,5cyclictest150-21pr/ttyS220:46:261
140215299362,5cyclictest131rcu_preempt20:29:593
1402152993621,12cyclictest1405175-21sshd20:06:153
140215299361,5cyclictest131rcu_preempt20:20:003
140215299361,3cyclictest131rcu_preempt19:50:003
140214799363,3cyclictest131rcu_preempt19:55:002
140215299353,3cyclictest131rcu_preempt20:33:153
140215299351,3cyclictest131rcu_preempt19:35:023
140215299351,3cyclictest131rcu_preempt19:35:013
140214799353,3cyclictest131rcu_preempt19:10:032
140214799353,3cyclictest131rcu_preempt19:10:022
140214299356,7cyclictest131rcu_preempt19:56:091
140215299344,3cyclictest131rcu_preempt19:10:163
1402152993427,4cyclictest1406761-21latency_hist20:40:023
140215299331,3cyclictest131rcu_preempt19:10:023
140215299331,3cyclictest131rcu_preempt19:10:023
140215299321,4cyclictest131rcu_preempt19:56:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional